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Abstract 


This  investigation  examined  the  preparedness  of 
Preliminary  Design  Review  (PDR )  and  Critical  Design  Review 
(CDR)  participants.  Background  and  opinion  data  were 
gathered  from  junior  and  senior  program  managers  and 
development  engineers  in  order  to  assess  their  perception  of 
PDR  and  CDR  purposes,  effectiveness/efficiencies,  training 
and  guidance. 

The  analysis  was  accomplished  by  sending  a  survey 
instrument  to  a  sampling  of  program  managers,  and 
development  engineers  thoughout  the  Air  Force  population 

o 

within  the  boarders  of  the  U.S.A. 

The  respondents  tended  to  be  in  agreement  with  the 
PDR/CDR  purposes  stated  in  Nil  Std  1521B. 

The  analysis  revealed  that  most  PDR/CDRs  are  not  as 
effective  as  -they  could  be.  The  primary  reason  is  the  lack 
of  knowledge  on  what  should  be  accomplished  by  the 
participants . 

Host  respondents  claimed  self  teaching  as  the  method  of 
^earning  their  preparation  for  PDR/CDRs.  Over  86.2  percent 
of  all  respondents  felt  initial  training  would  be  useful  and 


the  majority  indicated  6  to  12  months  of  acquisition  ,  /  AT 
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experience  should  be  required  before  participating  in  a 
PDR/CDR .  The  most  important  guide  for  PDR/CDR  preparation 
recommended  by  the  respondents  with  previous  PDR/CDR 
experience  was  the  Mil  Std  1521  and  Defense  System 
Management  College  (DSMC)  System  Engineering  Management 
Guide. 


PRELIMINARY  AND  CRITICAL  DESIGN  REVIEW 
PROCEDURES  EFFECTIVENESS 
AND  EFFICIENCY 

I .  Introduction 

Each  year  the  US  Air  Force  spends  millions  of  dollars 
on  technical  meetings.  Government  contractors  spend  a 
significant  amount  of  time  and  money  for  the  same. 

These  technical  meetings  range  from  Systems 
Requirements  Reviews  to  Production  Readiness  Reviews.  The 
two  meetings  of  interest  for  this  research  are  the 
Preliminary  and  Critical  Design  Reviews  (PDR,  CDR) .  These 
are  the  critical  technical  meetings  that  are  held  in  the 
Full  Scale  Development  (FSD)  phase  of  an  acquisition. 

Basically,  the  technical  meeting  is  a  tool  for 
evaluating  and  controlling  an  acquisition  program.  It  allows 
the  reviewers  to  focus  their  attention  on  the  design  process 
and  allows  them  to  review  design  concepts.  Specifically,  the 
PDR  is  concerned  with  reviewing  the  preliminary  design 
against  the  system  development  specification,  and  the  CDR  is 
concerned  with  reviewing  the  detailed  design  against  the 
draft  product  specification.  The  development  process 
requires  the  preliminary  design  requirements  be  satisfied 
before  the  program  begins  its  detailed  design,  and  that  the 
detailed  design  requirements  be  satisfied  before  a  program 
ente’-’j  into  the  fabrication  process  in  the  FSD  phase. 
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The  Military  Standard  Technical  Reviews  and  Audits  for 

Systems, Equipments, and  Computer  Programs  (Mil  Std  1521) 

defines  the  PDR  operationally  as: 

the  review  that  shall  be  conducted  for  each 
Configuration  Item  (CI>  or  aggregate  of  CIs  to: 

(1)  evaluate  the  progress,  technical  adequacy,  and 
risk  resolution  (on  a  technical,  cost,  and 
schedule  basis)  of  the  selected  design  approach, 

(2)  determine  its  compatibility  with  performance 
and  engineering  speciality  requirements  of  the  Cl 
development  specification,  (3)  evaluate  the  degree 
of  definition  and  assess  the  technical  risk 
associated  with  the  selected  manufacturing 
methods/processes,  and  (4)  estrblish  the  existence 
and  compatibility  of  the  physical  and  functional 
interfaces  among  the  Cl  and  other  items  of 
equipment,  facilities,  computer  programs,  and 
personnel (5:3) . 

The  CDR  is  defined  operationaly  as:  the 
review  that  shall  be  conducted  for  each  Cl  when 
the  detailed  design  is  essentially  complete  and 
its  purposes  are  to:  (1)  determine  that  the  detail 
design  of  th«  Cl  under  review  satisfies  the 
performance  and  engineering  speciality 
requirements  of  the  Cl  development  specifications, 

(2)  establish  the  detail  design  compatibility 
among  the  Cl  and  other  items  of  equipment, 
facilities,  computer  programs  and  personnel, 

(3)  assess  Cl  risk  areas  (on  a  technical,  cost, 
and  schedule  basis),  (4)  assess  the  results  of  the 
producibility  analysis  conducted  on  system 
hardware,  and  (5)  review  the  preliminary  product 
specifications  (5:3). 

The  Mil  Std  1521  is  applicable  to  all  technical  reviews 
and  audits  and  states  thr*t  each  review  shall  be  conducted  in 
accordance  with  its  contents  to  the  extent  specified  in 
program  contract  clauses.  Statement  of  Work,  and  Contract 
Data  Requirement  List  (5:1). 


General  Issue 


The  1982  Defense  Science  Board  Task  Force  on  Transition 
from  Development  to  Production,  formed  by  the  Undersecretary 
of  Defense  for  Research  and  Engineering  (USDRE) ,  found  as 
part  of  their  review  of  problems  within  the  acquisition 
process,  that  mbst  reviews  have  a  lack  of  direction  and  fail 
to  achieve  the  main  purposes  of  reviews,  i.e.,to  identify 
technical  risks  and  challenge  potentially  inadequate  system 
designs.  In  addition,  most  reviews  are  allotting  more  time 
for  tutorials  and  system  familiarization  than  for  design 
problems.  For  the  most  part  design  reviews  have  become  a 
time-consuming  exercise  contributing  little  to  the  assurance 
of  design  maturity  (6:  4-18).  The  Defense  Science  Board  Task 
Force  report  also  stated,  "In  the  acquisition  process,  first 
evidence  of  weapon  systems  problems  usually  becomes  apparent 
when  a  program  transitions  from  full-scale  development  into 
production."  (6:  2-1).  This  transition  point  is  within  the 
time  of  Development  Test  and  Evaluations  of  a  system,  and 
could  be  the  first  time  design  flaws  not  discovered  during  a 
PDR  or  CDR  ate  revealed. 

Mil  Std  1521  provides  for  specific  guidance  on  how  to 
conduct  design  reviews;  it  outlines  the  tasks  and 
responsibilities  of  both  the  contractor  and  the  Air  Force. 
The  Mil  Std  1521  describes  procedures  to  be  followed  before, 
during  and  after  each  review  session.  It  also  establishes 
the  appropriate  time  to  schedule  each  type  of  review.  Only 
after  the  successful  completion  and  approval  of  these 
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critical  reviews  should  a  program  enter  into  the  next 
acquisition  phase. 

However,  are  these  procedures  being  followed  or  are 
they  adequate?  This  research  addresses  this  question. 

Specific  Problem 

Four  specific  problems  with  Preliminary  and  Critical 
Design  Reviews  were  considered  :  1)  reviews  are  used  more  as 
tutorials  and  for  familiarizing  the  Air  Force  attendees  with 
the  system  hardware  than  for  design  problems,  i.e.  reviews 
often  become  a  forum  for  providing  an  overview  of  the 
overall  hardware  design  rather  than  an  in-  depth  technical 
assessment  of  design  maturity;  2)  most  Air  Force  attendees 
have  not  received  sufficient  prior  training  and  have  very 
limited  prior  experience  in  how  to  prepare  for  design 
reviews  or  how  to  conduct  themselves  at  these  review;  3) 
procedures  of  Mil  std  1521  are  not  being  followed;  4) 
meetings  are  not  efficient,  i.e.,  action  items  opened  and 
technical  problems  identified  are  minimal  as  compared  to 
problems  found  after  the  conclusion  of  the  design  review. 

Investigative  Questions 

The  investigative  questions  are  grouped  into  three 
major  areas  of  concern  for  this  research;  Mil  Standard  1521 
PDR/CDR  purposes,  PDR/CDR  effectiveness/efficiency,  PDR/CDR 
training  and  guidance. 
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Mil  Std  1521  PDR/CDR  Purposes,  The  three  investigative 
questions  are  as  follows: 

Question  1.  Are  acquisition  program  PDRs  and  CDRs 
conducted  in  accordance  with  the  purposes  stated  in  Mil  Std 
1521? 

Question  2.  Do  program  team  members  agree  with  the 
PDR/CDR  purposes  outlined  in  Mil  Std  1521? 

Question  3.  Do  PDR  and  CDR  experienced  individuals 
perceive  The  Mil  Std  1521  PDR/CDR  purposes  to  be  of  greater 
importance  than  do  those  without  PDR/CDR  experience? 

PDR  and  CDR  Ef feet iveness/Ef f iciancy.  The  thirteen 
investigative  questions  are  as  follows: 

Question  4.  What  approval  was  given  at  conclusion 
of  the  PDR  and  CDR? 

Question  5.  Did  the  PDR  and  CDR  provide  an 
adequate  evaluation  of  the  system  to  proceed  into  the  next 
acquistion  phase? 

Question  6.  Were  all  action  items  (AIs)  resolved 
before  the  PDR/CDR  approval  was  given? 

Question  7.  Were  any  AIs  left  open  at  the 
conclussion  of  the  PDR  and  CDR? 

Question  8.  Of  AIs  presented  at  the  PDR/CDR  were 
any  issued  previously  and  were  these  closed  prior  to  the 
PDR/CDR  completion? 

Question  9.  Should  all  AIs  be  resolved  before 
entering  a  PDR  or  CDR? 
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Question  10.  Were  there  slippages  in  the 
system/subsystem  design/development  schedule  and  CDP 
schedule? 

Question  11.  Should  the  design  be  complete  before 
approving  a  CDR? 

Question  12.  Were  major  design  problems  and 
modifications  presented  at  the  CDR  and  did  any  of  these 
exist  previously? 

Question  13.  Were  the  CDR  supporting  data  packages 
effective  in  supporting  the  system  review  (i.e.  complete, 
delivered  on  time)? 

Question  14.  Should  there  be  a  system  overview  or 
tutorial  as  part  of  a  PDR  and  CDR;  how  much  of  a  PDR/CDR 
should  be  devoted  to  system  overview  and  tutorials? 

Question  15.  What  correlations  exist  between  an 
individual's  opinion  of  the  amount  of  system  overview 
required  to  the  amount  conducted  on  the  last  PDR/CDR? 

Question  16.  Were  AF  participants  were  required  to 
state  their  individual  PDR/CDR  objectives? 

PDR  and  CDR  Training  and  guidance.  The  four 
investigative  questions  are  as  follows: 

Question  17.  What  type  of  PDR  and  CDR  training  and 
guidance  have  individual  team  member  had? 

Question  18.  Would  initial  PDR/CDR  training  be 

useful? 
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Question  19.  How  much  acquisition  experience 
should  an  individual  have  before  participating  in  a  PDR  or 
CDR? 

Question  20.  What  is  the  single  most  useful  guide 
for  PDR  and  CDR  procedural  guidance? 
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II.  Literature  Review 


Introduction 

What  i»  the  perceived  effectiveness  of  technical 
meetings  held  between  the  Department  of  Defense  (DoD) 
component  services  and  industry? 

Each  year  DoD  and  industry  spend  large  sums  of  money  to 
conduct  design  reviews.  However,  the  outcome  of  these 
reviews  is  not  always  perceived  to  be  effective.  However, 
the  outcome  of  these  reviews  is  not  always  perceived  to  be 
effective.  In  fact  the  Packard  Commission  (12:  66-67)  report 
stated  that  each  year  billions  of  dollars  are  spent  more  or 
less  efficiently. 

Technical  meetings  or  design  reviews  are  used  by  the 
DoD  component  services  as  a  means  for  evaluating  the 
progress  of  a  system's  design  and  development* 

This  literature  review  present  the  findings  of  a 
literature  search  on  technical  meetings  and  design  reviews. 
This  review  focuses  on  the  Air  Force  and  industry  comments 
of  meeting  purposes,  objectives,  procedures  and  problems. 

Definition 

A  design  review  is  a  formal,  completely  documented  and 
systematic  study  of  the  actual  status  of  a  design,  as 
opposed  to  its  forecast  status,  together  with  a  definition 
of  the  next  steps  to  be  taken.  According  to  Kitagawa 


(10:  212),  "A  design  review  is  sn  effective  method  for 
improving  the  reliability  of  a  product,  decreasing  the  cost 
and  reducing  the  development  time."  The  System  Program 
Of f ice/Bnglneer ing  Handbook  (1:  56)  states,  "Technical 
meetings  are  the  bread  and  butter  of  a  successful  program." 
In  contrast,  at  t)  what  a  technical  meeting  is,  the  Defense 
System  Management  College  (DSMC) ,  System  Engineering 
Management  Guide  (SEMG)  (3:  13-9)  states, "Design  reviews  are 
not  tutorials." 

Justification  of  the  Search  and  Review 

The  justification  of  this  research  is  to  ensure  that 
each  participant  at  a  review  has  a  complete  understanding  of 
what  is  required  of  himself,  and  to  identify  lessons 
lerrned,  so  that  the  Air  Force  gets  maximum  productivity  per 
dollars  spent  to  conduce  a  review. 

Discussion? 

Description.  All  design  and  development  is  some  form 
of  compromise  between  confliccing  requirements.  These 
conflicts  create  the  necessity  to  examine  performance 
characteristics,  reliabilit iy,  and  maintainability  of  a 
design,  and  to  relate  each  to  the  other  and  to  whole-life- 
cost  (13:16)  . 

Design  reviews  are  repetitive  activitities  and  are  to 
provide  a  representative  evaluation  of  the  on  going  design 
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to  management  and  anyone  elaa  involved  in  the  project.  Theae 
reviews  should  analyse  the  status  of  the  development  and 
decide  future  action  (13:16). 

Kitagawa  (19:212)  states: 

One  distinguishing  feature  of  the  design  review  is  its 
capability  to  make  positive  use  of  the  technical 
knowledge,  experience,  and  information  possessed  by 
specialists. 

If  there  is  an  inadequate  understanding  of  design  by 
the  reviewer,  it  will  be  impossible  for  him  to 
demonstrate  his  specialized  knowledge  and,  if  this 
occurs,  the  design  roview  will  not  produce  good 
results.  So  it  is  necessary  to  have  broad  knowledge  and 
valuable  information  that  can  be  of  use  in  a  design 
review  (19:212). 


Parnas  and  Clements  (11:252)  made  the  following  comment 
about  reviews: 

we  can  compare  the  projects'  achievements  to  those  that 
the  ideal  process  calls  for.  We  can  identify  areas  in 
which  we  are  behind  (or  ahead) .  Regular  review  of  the 
project's  progress  by  outsiders  is  essential  to  good 
management.  If  the  project  is  attempting  to  follow  a 
standard  process,  it  will  be  easier  to  review. 


Objectives.  Some  objectives  of  technical  meetings  and 
design  reviews  are: 

1)  confirm  that  designs  meet  requirements,  2)  uncover 
any  hidden  design  flaws,  3)  reduce  the  variety  of 
solutions  by  selection,  4)  foster  standar izat ion  of 
equipment  and  procedures,  5)  ensure  that  the  design  can 
be  produced  within  acceptable  tolerances  and  can  be 
controlled  within  specification,  6)  establish  and 
maintain  communications  across  interfaces,  7) focus  all 
activities  on  a  common  goal,  and  8) speed  up  development 
(13:21) . 

Design  reviews  are  conducted  within  stages  or  phases  of 
overall  programs.  In  industry  there  are  three  stages  of 
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review:  Driest  stage,  the  preliminary  review--held  at  times 
k>‘~  product  concept  studies,  2) second  stage,  the  intermediate 
reviewa-held  at  previously  fixed  decision  points,  3) third 
stage,  the  final  review-held  just  before  full  production 
commences  (13:21).  These  stages  correlate  with  the  Air 
Force  Reviews  as  defined  in  the  DSMC  SfiMG  (3:13-7):  1)  the 
System  Requirement  Review  is  held  at  the  beginning  of  a 
program  startup,  2)  the  System  Design  Reviews  are  he; i  at 
pre-established  time  intervals,  3)  the  final  review  of  a 
design  phase  is  held  at  the  ond  of  the  full  scale 
development  phase,  just  prior  to  releasing  the  design  to 
production  go-ahead  (3:13-7). 

Purpose.  Design  reviews  should  he  used  to  evaluate 
trade  offs  between  performance,  cost,  schedule  and 
supportability.  In  addition,  design  reviews  allow  the 
Government  to  overview  the  complete  system  design  and 
evaluate  its  capability  to  satisfy  total  mission 
requirements.  The  design  review  is  to  search  out  design 
weaknesses  or  faulty  designs  (3:13-6).  The  cost  of  any 
engineering  design  changes,  especially  in  the  later  stages 
of  a  program,  are  usually  very  large.  It  is  imperative,  in 
particularly  large  programs,  that  formalized  design  reviews 
be  established  early  (13:21). 

The  major  purpose  of  a  technical  meeting  is  to  review 
engineering  design  progress  toward  the  final  design 
specification  (13:21). 


The  general  purposes  of  PDR's  and  CDR'a  ace  described 
in  APR  800-14  and  AFSCP  800-3.  These  documents  refer  to  the 
Mil  Std  1521  for  specific  detailed  procedures  a  program 
should  follow  for  PDR  and  CDR. 

In  general,  the  purpose  of  a  PDR  is  to  evaluate  the 
design  concepts  the  contractor  intends  to  use  for  each  Cl  to 
meet  the  allocated  baseline  requirements.  The  purpose  of  a 
CDR  is  to  evaluate  the  detailed  design  (drawing,  flow 
charts)  of  each  Cl  to  see  if  it  will  be  able  to  achieve  the 
allocated  baseline  requirements  (15:214).  A  successful  PDR 
is  required  for  each  Cl  prior  to  proceeding  into  detail 
design  (4:4-4).  The  contractor  starts  detail  design  control 
with  CDR  (15:214) . 

The  Air  Force  Regulation  800-14  stresses  the 
requirement  for  technical  control,  technical  task 
required  to  progress  from  an  operational  need  or 
requirement  to  the  development  and  operation  of  the 
system  by  the  user.  Formal  technical  control  is 
accomplished  by  technical  reviews  at  discrete 
milestones  (4:4-3) . 

The  completeness  of  the  reviews  provide  the  basis 
for  rendering  decisions  furthering  the  course  of  the 
program  to  ensure  that  the  system  design  integrity  is 
maintained,  technical  deficiencies  are  isolated,  and 
necessary  changes  are  identified  promptly  with  formal 
technical  reviews  procedures  specifically  detailed  in 
MIL-STD-499A  (USAF)  and  MIL-STD-1521  (USAF)  (4:4-3). 


In  addition  to  the  requirements  outlined  in  AFR  800-14 
the  Air  Force  Systems  Command  Pamphlet  800-3  (AFSCP  800-3) 
describes  the  purposes  of  technical  meetings  as  to  review 
the  integrated  engineering  and  technical  deviation  of 
engineering  efforts  periodically  to  determine  the  technical 
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adequacy  of  contractor  afforta  in  meeting  system 
requirement*  (9:8-4). 


AFSCP  808-3  (9:8-4. 5)  outlines  some  of  the  major  items 
to  consider  when  planning,  conducting,  or  participating  in 
formal  reviews  as: 

a.  Assigning  contractually  binding  action  items  for 
disposition  of  non  conformances  identified. 

b.  Availability  of  detailed  documentation  to  assess 
contractor  progress  in  developing  specifications. 

c.  Reviewing  engineering  documentation  to  ensure  that 
it  facilitates  synthesis  and  integration  of  intrasystem 
items. 

d.  Understanding  the  types  of  decisions  that  can  be 
used  to  establish  the  adequacy  and  accuracy  of  the 
design  review. 

(1)  Unqualified  approval  -  specify  complete 
agreement. 

(2)  Approval  w/contingent  action  items  -  used 
when  the  review  is  not  considered  accomplished 
until  satisfactory  completion  of  actions. 

(3)  Approval  with  deviation  -  used  when  it  is 
in  the  interest  of  the  program  to  award  limited 
approval  and  protect  program  schedules  pending 
completion  of  future  engineering  as  indication  by 
action  items. 

(4)  Disapproval  -  used  when  review  is 
unsatisfactory  or  generally  inadequate.  A  new 
review  must  be  conducted  as  a  result  of 
disapproval . 


The  personnel  to  participate  in  design  reviews  should 
possess  program  expertise.  The  same  personnel  should 
participate  in  reviews  throughout  the  acquisition  process. 
This  ensures  consistent  technical  expertise  in  evaluating 
contractor  engineering  efforts. 

Using  and  supporting  commands  should  participate  to 
assist  the  program  office  but  not  give  specific  direction. 
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Deficiencies  should  be  isolated  at  the  earliest  point 
in  time,  and  necessary  changes  should  be  identified  promptly 
( AFSC  8.5). 

AFSCP  800-3  states  a  successful  PDR  is  required  before 
each  Cl  can  procede  into  the  detailed  design;  and  the 
successful  completion  of  CDR  is  required  for  each  Cl  before 
commitment  of  the  design  to  production  (9:8.5). 

Procedures .  Some  initial  procedures  to  follow  prior  to 
and  during  meetings  are:  1)  Identify  problems  early  from  the 
data  package  submission  and  attempts  made  to  clarify  them  on 
an  individual  basis;  2)  Avoid  presenting  major  p~  >blem 
surprises  (3:13-9);  3)  Establish  routine  communication 
channels  with  the  contractor;  4)  Have  effective  meetings 
frequent  enough  to  have  a  greater  probability  of  uncovering 
design  errors;  5)  Have  the  backing  of  top-level  management; 
6)  Have  the  necessary  data  available  in  order  to  have  a 
quality  review  (8:70);  7)  Work  the  meeting  agenda  (16:10); 

8)  Motivate  participants  before  the  start  of  meetings,  in 
terms  of  commitments  and  ask  participants  to  verbalize  their 
plans  for  the  meeting. 

The  supporting  data  packages  should  be  received  from 
the  contractor  and  reviewed  before  the  meeting.  This  data 
package  should  contain  applicable  engineering  drawings, 
specifications  and  reports.  According  to  Kitagawa  (10:214), 
"Good  use  of  data  packages  could  point  out  more  than  30%  of 
product  failures  due  to  misdesign."  Agenda  Items  which  may 
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meet  with  controversy  or  objections  should  be  placed  at  the 
end  of  a  long  agenda  of  a  meeting  to  ensure  other  items  will 
get  their  appropriate  coverage.  With  motivated  participants 
the  result  should  be  a  better  quality  performance  of 
reviewers  (16:9) . 

Some  recommended  procedures  for  technical  meetings  are 
as  follows: 

1.  Meetings  should  be  held  at  the  contractor  facility 
to  ensure  the  technical  experts  required  will  be  available. 

2.  At  the  end  of  each  meeting  day  personnel  should 
caucus  to  evaluate  action  items.  Decisions,  agreements,  and 
approved  action  items  should  be  recorded  and  signed  by  both 
the  Government  and  the  contractor  at  the  end  of  each  day. 
Each  action  item  should  be  assigned  two  responsible 
individuals  (one  Govenment,  one  contractor)  and  required  to 
respond  by  an  established  date  in  order  to  close  the  action 
item  (3:13-9) . 

3.  At  the  conclusion  of  the  review  a  summary  of  action 
items  taken  should  be  prepared  for  presentation.  Issues, 
questions,  agreements,  and  action  items  must  be  documented 
in  minutes  and  assigned,  tracked,  and  coordinated  with 
appropriate  participants  through  the  close  out  or  completion 
of  the  action  item  (3:13-8). 

4.  Evaluate  precisely  and  in  detail  to  find  the 
influences  of  over-designed  or  manufacturing  to  close 
tolerances  at  high  cost-  Determine  if  the  cost  provides 
significantly  better  performance  and  reliability  (QA/RM:17). 


In  addition  Welsch  (17:59)  states, 

"All  designs  should  be  reviewed  by  qualified  reviewers 
and  time  and  resources  should  be  made  available  to  do 
thorough  reviews.  If  not,  the  project  management  could 
be  forced  to  take  the  time  and  spend  10  or  100  times 
the  money  if  problems  surface  during  construction  'pay 
me  now  or  pay  me  later'." 

In  controlling  changes  the  acquisition  team  should 
always  ask  the  question,  "Is  the  change  really  necessary?" 
"What  are  the  consequences  ‘if  the  change  is  not  made?" 

These  changes  should  not  be  allowed  to  accumulate.  If 
the  contractor  does  not  cooperate  with  timely  proposals,  his 
right  to  proceeds  with  that  portion  of  the  work  can  be 
withdraw.:  just  long  enough  to  get  his  attention  and  no 
longer  (17:59)  . 

Also,  the  team  size  should  be  held  to  a  minimum  number 
of  participants  to  adequately  cover  the  areas  of  the  review 
so  that  excessive  time  is  not  used  in  dialogue  between 
attendees  (3:13-9).  It  is  important  to  ensure  the  review 
team  leader  is  an  experienced  member,  experienced  in 
research  and  development,  for  the  given  stage  of  a  project. 
Then  in  the  manufacturing  stage  he  should  be  replaced  by  a 
production  manager.  This  provides  the  proper  experience  at 
the  appropriate  stage  of  the  program  (13:21). 

Both  AFR  800-14  and  AFSCP  800-3  refer  to  the  Mil  Std 
1521  for  the  specific  details  of  the  contractor's  and 
procuring  activity's  role  in  technical  meetings. 

The  contractor  is  responsible  for  establishing  the 
time,  place  and  agenda  in  accordance  with  the  master 
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milestone  schedule,  subject  to  procuring  activity 
coordination.  It  should  be  accomplished  sufficiently  in 
advance  to  allow  for  adequate  preparation  for  the  meeting. 

In  addition,  the  contractor  should  prepare  for  each 
review  in  sufficient  detail  consistent  with  the  scope  and 
magnitude  of  the  review;  designate  a  co-chairperson  for 
each;  record  minutes  consisting  of  significant  questions  and 
answers,  action  items,  deviations,  conclusions  and 
recommended  courses  of  action  resulting  from  presentations 
or  discussions.  Recommendations  not  accepted  should  also  be 
recorded  together  with  the  reason  for  non-acceptance.  All 
action  items  should  be  cJ early  recorded  in  the  minutes  and 
identify  whether  procurring  activity  and/or  contractor 
action  is  required  for  the  resolutions  (5:6). 

Contractor  shall  be  required  to  provide  the  necessary 
resources  and  material  to  perform  the  review  effectively 
(5:5). 

Procurring  activity  role  is  to  review  the  minutes 
(daily)  and  ensure  these  reflect  all  significant  procuring 
activity  inputs.  Provide  formal  acknowledgement  to  the 
contractor  of  the  accomplishment  of  each  review  by  notifying 
the  contractor  of: 

Approval  -  indicate  review  was  satisfactorily  completed. 

Contingent  approval  -  indicate  the  review  is  not 
considered  accomplished  until  the  satisfactory  completion  of 
resultant  action  items. 


Disapproval  -  indicate  that  the  review  was  seriously 
inadequate  (5:7). 

Problems.  The  major  problem  is  there  are  no  formally 
established  methods  on  how  to  structure  or  conduct  technical 
meetings  (ASD:56) .  In  addition,  many  of  the  Air  Force  review 
team  leaders  are  inexperienced  in  the  best  approaches  to 
conducting  technical  meetings. 

In  the  past  it  was  easy  to  overlook  important  design 
problems  in  the  design  planning  stage,  and  major  design 
modifications  were  often  done  during  the  detailed  design 
stage.  This  practice  led  to  design  flaws,  cost  and  schedule 
problems  (10:212). 

Formal  reviews,  conducted  through  a  committee,  consist 
of  representatives  from  engineering,  marketing, 
manufacturing,  quality  control,  and  purchasing.  This  team 
make-up  led  to  the  reviewers  sometimes  adopting  the  view 
that  the  design  reviews'  function  is  to  veto  designs  rather 
than  to  provide  information  and  ideas  for  improving  the 
designs  (2:96) . 

Meetings  held  too  frequently  can  cause  the  contractor 
to  spend  a  great  deal  of  time  preparing  for  meetings  at  the 
expense  of  doing  the  actual  work  (1:56). 

Some  specific  examples  of  problems  encountered  in 
industry  directly  attributed  to  a  lack  of  good  design  review 
practices  are  as  follows: 

1.  An  audit  of  construction  change  orders 

disclosed  that  936  out  of  2,035  (46%)  of  the 
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construction  change  orders  reviewed  involving  $67.4 
million  (17:57-58). 

2.  The  cost  of  six  projects  for  construction  of 

air  traffic  control  tower  facilities  increased  by  more 
than  $760,000  (17:58) . 

3.  Total  manhours  of  the  data  packages  were  5-10% 
of  the  total  design  manhours  and  the  manhours  of  the 
meetings  were  10-50%  the  total  review  manhours 
(10:213) . 

4.  A  change  order  for  $7  million  to  clean,  repair 
and  repaint  16  fuel  storage  tanks  (17:50). 

5.  Studies  show  that  upwards  of  55%  of  all 
software  errors  are  introduced  in  the  early 
[Requirements  Definition]  phase  but  only  15%  of  those 
errors  were  found  and  corrected  by  the  end  of  this 
phase  (7:42). 

6.  Operations  and  maintenance  software  costs  for  a 
system  with  an  expected  useful  life  of  8  to  20  years 
will  amount  to  250  to  500%  of  the  development  costs;  of 
that  amount,  only  about  15%  goes  into  correcting  the 
modifications  to  meet  new,  or  missed  requirements 
(7:47). 

Some  additional,  more  general,  problems  with  reviews 


are: 

1.  The  number  of  design  review  meetings  and  the 
total  manhours  devoted  to  a  review  were  of  little 
importance,  but  the  level  of  technology  was  of  great 
importance.  In  other  words,  the  products  for  which 
reviews  were  very  effective  were  most  often  those  for 
which  concrete  objectives  had  been  set,  for  which 
preparation  had  been  made.  Products  for  which  reviews 
were  not  very  effective  were  most  often  those  for  which 
level  objectives  and  level  predictions  could  not  be 
adequately  established  because  the  necessary 
information  and  data  were  unavailable  (10:214). 

2.  If  the  data  packages  which  serve  to  clarify  the 
design  process  were  not  available  at  the  design  stage, 
design  review  would  not  always  contribute  to  the 
reduction  of  design  changes  after  release  of  the  design 
to  manufacturing  (10:212). 

The  results  of  design  reviews  are  evaluated  based  on 
number  of  established  action  items,  frequency  of  design 
changes  and  past  review  investigations  (10:213). 

The  Defense  Science  Board  Task  Force  (6:16:4.8-4.9) 
describes  four  traps  that  a  review  can  have  and  the  benefits 
if  the  best  design  review  practices  are  followed  and  the 
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consequence  if  not  followed.  These  steps,  benefits  and 
consequences  are: 

1.  When  program  review  formats  and 

"Best"  practices  are  used: 

technical  balance  can  be  maintained 
between  management  and  design. 

or  "Current"  practices  are  used: 

reviews  over  staffed  with  management 
personnel  and  management  status  is  reviewed. 

2.  When  the  review  is  keyed  to  program  milestones  and 

"Best"  practices  are  used: 

design  maturity  can  be  determined 

or  "Current"  practices  are  used: 

reviews  are  success-oriented,  not  a  technical 
evaluation.  Risk  is  not  identified  or 
assessed  and  design  deficiencies  are  not 
identified. 

3.  When  the  review  is  focused  on  the  design  and 

"Best"  practices  are  used: 

the  design  will  fullfill  all  specified 
requirements . 

or  "Current"  practices  are  used: 

analyses,  assumptions,  and  processes  are  not 
reviewed;  trade-off  studies,  underlying  data, 
and  risk  assessments  are  not  presented.  Thus, 
design  is  not  influenced  by  all  analytical 
activities. 

4.  When  design  reviews  are  held  informally  and 

"Best"  practice  are  used: 

design  baselines  can  be  certified. 

or  "Current"  practices  are  used: 

design  review  actions  are  not  reported  to 
management  and  a  formal  report  with 
appropriate  action  items  is  not  prepared. 

So,  total  system  requirements  are  not  met 
(6:4. 8-9) . 
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SUMMARY 


Reviews  are  important,  especically  the  Preliminary  and 
Critical  Design  Reviews.  These  are  the  most  significant 
reviews  of  an  acquisition  program.  However,  if  these 
reviews  are  not  conducted  in  a  structured  format  with 
specific  procedures  and  the  participation  of  appropriately 
trained  and  experienced  personnel,  the  time  and  money 
expended  by  the  Air  Force  and  the  contractor  could  be 
considered  wasted.  In  addition,  the  end  item  delivered 
could  very  easily  exceed  original  cost  and  delivery  schedule 
and  still  not  meet  the  minimum  design  requirements 
established  at  the  beginning  of  a  program. 
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III.  Methodology 


introduction 

This  chapter  describes  the  methodology  used  to  collect 
and  analyze  the  data  required  to  answer  all  the 
investigation  questions  posed.  The  data  collection 
instrument  will  be  discussed  followed  by  discussions  on 
sample  population,  sample  plan,  data  analysis,  assumptions 
and  limitations. 

Data  Collection  Instrument 

A  mail  survey  was  used  to  gather  background  information 
and  opinion  data  necessary  to  complete  this  research 
(Appendix  A) . 

The  survey  measures  perceptions  and  attitudes  toward 
PDR  and  CDR  effectiveness  in  evaluating  system  design  and 
development  progress.  It  questioned  if  established  Air  Force 
standard  procedures  are  being  followed  and  how  effective 
these  are  perceived  by  Air  Force  program  managers  and 
development  engineers.  The  survey  was  designed  to  gather 
sufficient  data  to  determine  if  perceived  purposes  of  PDRs 
and  CDRs  correlate  to  purposes  satisfied  on  the  individual's 
most  current  program,  Tho  data  was  analyzed  and  evaluated  to 
determine  PDR  and  CDR  effectiveness,  and  to  determine 
whether  or  not  there  is  a  need  for  improved  PDR  and  CDR 


procedure  guidance.  The  survey  instrument  was  conducted  in 
May  to  June  1987. 

The  first  nino  questions  of  the  survey  provide 
background  data  for  each  of  the  individual  respondents. 

Questions  10  through  14  provide  the  respondents 
opinions  on  agreement  with  the  purposes  of  PDR  outlined  in 
Mil  Standard  1521. 

Questions  17  through  22  provide  the  respondents  opinion 
on  agreement  with  the  purposes  of  CDR  outlined  in  Mil 
Standard  1521. 

Questions  25  through  29  provide  the  respondents 
perception  of  how  well  the  Mil  Std  1521  PDR  purposes  of  the 

last  PDR  he  attended  were  satisfied. 

•  * 

Questions  33  through  37  provided  the  respondents 
perception  of  how  well  the  Mil  Std  1521  CDR  purposes  of  the 
last  CDR  he  attended  were  satisfied. 

Questions  14,16,24,30,38  provided  the  respondents 
opinion  on  system  overview  and  tutorial  as  these  relate  to 
purposes  of  PDR  and  CDR. 

Questions  43,49  and  53  through  54  provide  the 
respondents  opinion  on  PDR  and  CDR  guidance  and  training. 

The  survey  questions  are  summarized  in  Table  1  and 
matched  to  the  specific  investigative  questions. 


TABLE  1 


Survey  Structure 

Investigative  Survey 

Question _ Question _ 

1  25-29  (PDR) 

33-37  (CDR) 


2  lfl-13  (PDR) 

17-22  (CDR) 


3 

6 ,  10-13 
6,  17-22 

(PDR) 

(CDR) 

4 

32,41 

5 

6,  56 

6 

31,  39 

7 

45 

8 

46,  47 

9 

6,  9,  15 
6,  9,  23 

(PDR) 

(CDR) 

10 

44,  52 

11 

6,  9,  42 

12 

40,  48 

13 

50,  51 

14 

6,  9,  16 
6,  9,  24 

(PDR) 

(CDR) 

15 

14,  30  (PDR) 
12,  38  (CDR) 

16 

49 

Ares  of  Concern 


Mil  St4  1521 

PDR/CDR 

Purposes 


PDR/CDR  Effect¬ 
iveness/Effici¬ 
ency 
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Table  1  Continued 

17  9,  43  PDR/CDR  Training 

and  Guidance 

18  9 ,  53 

19  6,  9 ,  54 

28  9 ,  55 

Sample  Population 

The  population  included  all  active  duty  Air  Force 
officers  stationed  within  the  contintent  of  the  United 
States  of  America  with  a  grade  from  0-2  to  0-5  who  are 
currently  holding  Duty  Air  Force  Speciality  Code  identifiers 
of  2716,  2724  (Acquisition  Management  poitions)  and  2816, 
28X5  (Development  Engineering  positions). 

Based  on  personnel  manpower  listings  provided  by,  there 
are  currently  614  officers  with  a  DAFSC  of  2716,  921 
officers  with  a  DAFSC  of  2724,  776  officers  with  a  DAFSC  of 
2816  and  3243  officers  with  a  DAFSC  of  28Xb.  Table  2  shows 
the  population  size  for  each  subgroup.  The  table  data  was 
generated  from  an  Atlas  Data  Base  Statistic  inquiry  on  15 
April  1987. 

A  92.5  percent  confidence  level  was  selected  for  this 
research.  A  simple  random  sampling  was  used  in  selecting 
sampling  subgroups.  The  subgroups  matrix  the  four  DAFSC 
(2716,2724,2816  and  28X5)  by  ranks  (0-2, 0-3, 0-4  and  0-5). 
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TABLE  2 

Rank  by  DAFSC  Matrix 


2716 

DAFSC 

2724 

2816 

28X5 

0-5 

371 

— 

351 

— 

0-4 

238 

38 

424 

207 

RANK 

0-3 

5 

462 

1 

1608 

0-2 

— 

421 

—  - 

1428 

Total 

614 

921 

776 

3243 

The  following  formula  was  used  to  determine  the  sample 
size  necessary  for  each  subgroup  to  meet  the  desired 
confidence  level  of  92.5  percent: 


n  -  (N (i*2) *p(l-p)  1/t  (n-1)  *(d“2)  +  (z*2)*p(l-p)  ]  (eg.  1) 


where:  n  *  sample  size 

N  «  population  size  from  each  subgroup 
P  «  maximum  sample  size  factor  (.50) 

D  *  desired  tolerance  (.075) 

Z  ■  factor  of  assurance  (1.44)  for  92.5  percent 
confidence  level 


Only  company  grade  officers  (0-2  to  0-3)  were  entered 
in  the  data  base  for  company  grade  DAFSCs  of  2724  and  28X5, 
and  only,  field  grade  officers  (0-4  to  0-5)  were  entered  in 
the  data  base  for  the  field  grade  DAFSCs  of  2716  and  2816. 
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The  sample  size  necessary  from  each  subgroup  to  meet 
the  92.5  percent  confidence  level  was  57  0-4s,  88  0-5s 
(DAFSC  2716),  70  0-2s,  76  0-3s  (DAFSC  2724),  82  0-4s,  68 
0-5s  (DAFSC  2816)  and  72  0-2s,  81  0-3s  (DAFSC  28X5). 

Data  Analysis 

FAS  statistical  package  was  used  to  analyze  data  where 
appropriate.  Frequencies  and  percentages  of  each  response 
were  calculated  for  each  survey  question.  Crosstabulations 
of  the  numbers  and  percentages  of  each  possible  response  to 
each  survey  question  were  calculated  for  each  of  the  sample 
population  subgroups. 

Chi-squared  test  of  categorical  data  involves  taking  a 
sample  from  a  single  population  and  classifying  each 
individual  with  respect  to  two  different  categorical  factors 
(such  as  religious  preference  and  political  party 
registraion) .  The  null  hypothesis  in  this  situation  is  that 
the  two  factors  are  independent. 

The  critical  value  to  test  against  is  dependent  upon 
the  degrees  of  freedom  and  the  selected  alpha  value. 

For  this  research  an  alpha  value  of  0.075  was  selected 
to  be  consistent  with  the  research  confidence  level  of  92.5 
percent . 

The  SAS  program  provided  chi  square  probability  values 
for  each  of  the  cross  tabulatioin  tables.  A  computed  chi 
square  probability  value  smaller  than  0.075  indicates  the 
two  factors  are  not  independent  of  each  other. 
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The  chi-square  test  was  used  with  investigative 
question  10. 

SAS  Proc  Corr  was  used  to  compute  the  correlation 
factor.  Previous  studies,  as  discussed  in  the  AFIT  ORSC  542 
class.  Management  and  Behavior  in  Organizations,  have 
indicated  a  correlation  factor  of  0.25  is  good  when 
addressing  opinions  and  attitudes  of  people.  A  correlation 
factor  of  0.4  is  very  good  and  0.38  or  greater  is 
significant  (14) . 

The  results  of  the  frequency  tabulations,  chi-square 
goodness  of  fit  test,  correlations  and  crosstabulations 
that  correspond  to  investigative  questions  are  presented  in 
chapter  four. 

Assumptions  and  Limitations 

Assumptions.  This  research  assumed  that  the 
respondends  answered  all  questions  accurately  and  that  their 
own  attitudes,  opinions  and  perceptions  are  reliable  and 
valid. 

Limitations .  No  attempt  has  been  made  to  generalize 
the  results  Air  Force  wide.  A  large  part  of  any  PDR  and  CDR 
part’  upant  population  include,  not  only  Air  Force  military, 
but  t  a  civil  service  work  force.  This  large  population 
(Civil  Service)  was  not  surveyed  due  to  the  time  required 
for  civil  service  survey  approval. 
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Before  any  generalization  could  be  made  response  data 
from  civil  service  program  managers  and  development 
engineers  carreer  fields  would  have  to  be  included. 

Other  limitations  concern  the  accuracy  involved  in 
measuring  attitudes,  opinions  and  perceptions. 

Summary 

This  chapter  described  the  methodolgy  that  was  used  in 
this  research  project.  Chapter  four  discusses  the  results  of 
this  research  and  chapter  five  summarizes  the  findings  and 
provides  recommendations. 
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Introduction 


This  chapter  provides  the  results  of  the  survey  in 
answering  the  investigative  questions.  The  results  are 
presented  in  tabular  form  and  represent  frequency  of 
response  to  each  question.  Addressing  some  of  the  ' 
investigation  questions  involved  compiling  and  analyzing 
data  on  more  than  one  survey  question. 

Survey  Response  Rates.  Tables  3. a  and  3.b  contain  a 
summary  of  the  number  and  percentage  of  responsdents  to  the 
survey. 

TABLE  3. a 

Frequency  of  Response 
by  DAFSC  Against  Rank 

DAFSC 


2716  2724  2816  28X5 


Rank 

LtCol 

31 

_  _ 

15 

„  _ 

Ma  j 

24 

— 

15 

— 

Capt 

— 

24 

-  — 

83 

TABLE  3.b 

Percentage  of  Response 
by  DAFSC  Against  Rank 


Rank 

2716 

DAFSC 

2724 

2816 

28X5 

LtCol 

41.9 

— 

28.8 

— 

Ma  j 

50.0 

— 

23.8 

— 

Capt 

34.8 

34.6 

lLt 

— 

38.1 

— 

31.0 

The  response  rate  to  the  survey  questionaire  were 
poorer  than  expected.  The  survey  questionaire  was  sent  to 
790  individuals  with  only  271  returned.  The  92.5  percent 
confidence  level  set  for  this  research  could  not  be 
supported  in  the  subgroups  with  the  exceptions  of  the  DAFSC 
28X5  0-3  and  0-2  subgroups. 

The  subgroups  could  support  a  greater  than  85  and  less 
than  90  percent  confidence  level.  Three  subgroups  were 
eliminated  from  the  analysis,  DAFSC  2716  0-3s,  DAFSC  2724 
0-4s ,  DAFSC  28X5  0-4s.  DAFSC  2716  and  2816  are  field  grade 
positions  and  are  normally  supported  with  field  grade  ranks 
of  0-4  and  0-5;  0-3  would  be  unusual.  DAFSC  2724  and  28X5 
are  company  grade  positions  and  normally  support  the  company 
gade  ranks  of  0-1,  0-2  and  0-3;  0-4  would  be  unusual. 

This  section  will  address  each  category  of  survey 
question.  There  are  four  categories  of  survey  questions 
consisting  of  Background  information.  Mil  Standard  1521 
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PDR/CDR  purposes,  PDR/CDR  effectiveness/efficiency  measures 
and  PDR/CDR  draining  and  guidance.  The  format  for  all 
findings  and  analysis  of  data  pertaining  to  the 
investigative  questions  is  presented  in  this  section  as 
follows: 

1) .  Investigative  question 

2) .  Discussion 

3) .  Findings 

Background  Information 

Tables  4  through  9  cover  the  category  of  background 
information. 

Table  4  shows  the  percentage  of  total  respondents  by 
rank.  Over  40  percent  were  of  the  rank  of  0-3. 


TABLE  4 

Current  Military 

Rank 

Rank 

#  of 

Respondents 

%  of 

Respondents 

LtCol 

45 

17.5 

Ma  j 

40 

15.5 

Capt 

104 

40.5 

lLt 

68 

26.5 

Total 

257 

100.0 

32 

I 

I 
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Table  5  shows  the  acquisition  phase  the  respondents 
were  most  familiar  with.  Forty-nine  percent  were  most 


familiar  with  Full  Scale  Development  phase  of  acquisition. 

Acquisition 

TABLE  5 

Phase  Most  Familiar 

Phase 

Concept  Exporation 

*  of 

Respondents 

%  of 

Respondents 

10.8 

Demonstration/Validation 

64 

25.5 

Full  Scale  Development 

124 

49.4 

Production 

3C 

14.3 

Total 

251 

100.0 

Table  6  shows  the  number  of  years  of 

acquisition 

experience  the  respondents 

have.  Thirty-three  percent  of  the 

respondents  claimed  3-4  years  experience. 

Eighty-six  percent 

of  the  respondents  had  experience  ranging 

from  0  to  10 

years. 

TABLE  6 

Years  of  Acquisition  Experience 

Years 

#  of 

Respondents 

%  of 

Respondents 

0-2 

55 

21.4 

3-4 

85 

33.1 

5-7 

53 

20.6 

8-10 

29 

11.3 

11  -  14 

15 

5.8 

>  15 

20 

7.8 

Total 

257 

100.0 
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Table  7  shows  the  respondents  area  of  education.  The 
results  revealed  three  main  areas,  51.2  percent  held  a 
technical  bachelors  degree,  22.7  percent  held  a  technical 
masters  degree  and  21.0  percent  held  a  non  technical  masters 
degree. 


TABLE  7 

Education  Area 


Degree  Area 


#  of 

Respondents 


%  of 

Respondents 


Technical  Bachelors  135  51.2 

Non  Technical  Bachelors  10  4.0 

Technical  Masters  57  22.7 

Non  Technical  Masters  53  21.0 

Technical  Doctoral  3  1.2 

Total  252  100.1 


Table  8  shows  the  Duty  Air  Force  Specialty  Code  (DAFSC) 
identifier  of  each  repondent .  Thirty-two  percent  were  field 
grade  (2716,2816)  and  66.8  percent  were  company  grade 
(2724,28X5)  positions.  Thirty-seven  percent  were  project 
managers  (2716,  2724)  and  62.5  percent  were  development 
engineers  (2816,28X5) .  The  largest  DAFSC  group  consist  of 
28X5  for  47.7  percent  of  the  respondents. 


Duty 

TABLE  8 

AF  Specialty  Code 

DAFSC 

*  of 

Respondents 

%  of 

Respondents 

2716 

46 

18.0 

2816 

38 

14.8 

2724 

49 

19.1 

28X5 

122 

47.7 

Total 

255 

99.6 

Table  9  shows 

the 

percentage  of  all  respondents 

having 

participated  in  PDRs  and  CDRs  against 

their  DAFSC.  The  range 

of  participation  was  0 

to  50  for  PDRs 

and  CDRs.  Seventy-f our 

percent  of  all  respondents  had  participated  in  a  PDR 

and 

72.4  percent  had  participated  in  a  CDR 

• 

PDR/CDR  by 

TABLE  9 

DAFSC  Participation 

Matrix  (in  %) 

DAFSC 

PDR 

CDR 

2716 

78.26 

73.91 

2816 

80.56 

80.56 

2724 

68.03 

68.75 

28X5 

67.75 

68.85 
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Military  Standard  1521  POR/CDR  Purposes 


Tablet  10  through  39  covert  the  category  of  Mil 
Standard  1521  PDR/CDR  purposet. 

Inveatigative  gueationa  one  through  three  poted  in 
chapter  one  art  addressed  here. 


Inveatigative  Question  la.  Are  acquisition  program 
PDRs  conducted  in  accordance  with  the  purposes  stated  in  Mil 
Std  1521? 

Discussion.  Survey  questions  25  through  29  were 
the  stated  purposes  of  a  PDR  outlined  from  Mil  Std  1521.  The 
questions  were  phrased  to  gain  the  repondents  opinion  on 
whether  these  PDR  purposes  were  satisfied  on  the  last  PDR 
the  individual  attended. 

Survey  Question  25; 

The  review  adequately  covered  the  technical 
adequacy  of  the  selected  design  approach. 


Survey  Question  26: 

The  review  adequately  covered  the  risk 
resolution  (on  a  technical,  cost,  and 
schedule  basis)  of  the  selected  design 
approach. 


Survey  Question  27: 

The  review  adequately  covered  the  design 
approach  capability  in  meeting  the 
performance  and  engineering  specialty 
requirements  of  the  Cl  development 
specification. 
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Survy  Question  28 1 

The  review  adequately  coveted  the  technical 
risk  associated  with  selected  manufacturing 
methods  and  processes. 

Survey  Question  29* 

The  review  adequately  covered  the  physical 
and  functional  interfaces  among  the  Cl  and 
other  equipment,  facilities,  computer 
programs,  and  personnel. 

Findings.  Seventy-eight  percent  of  all 
respondents  range  in  agreement  from  moderate  to  strong  with 
the  stated  purpose  of  survey  question  25. 

Fifty-one  percent  of  all  respondents  range  in  agreement 
from  moderate  to  strong  with  the  stated  purpose  of  survey 
question  26. 

Sixty-one  percent  of  all  respondents  range  in  agreement 
from  moderate  to  strong  with  the  stated  purpose  of  survey 
question  27. 

Only  37.5  percent  of  all  respondents  range  in  agreement 
from  moderate  to  strong  with  the  stated  purpose  of  survey 
question  28,  38.4  percent  neither  agreed  nor  disagreed  and 
32.1  percent,  moderate  to  strongly  disagreed. 

Slightly  over  half,  52.2  percent,  range  in  agreement 
f’tom  moderate  to  strong  with  the  stated  purposes  of  survey 
question  29. 
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TABLE  18‘ 

Survey  Question  25 t  Lest  FDR  Technical  Adequacy 


Response 

1  of 

Respondents 

«  of 

Respondents 

Strongly  Agree 

33 

18. • 

Moderately  Agree 

111 

68.7 

Neither  Agree  or  Disagree 

13 

7.1 

Moderately  Disagree 

23 

12.8 

Strongly  Disagree 

_ 3 

1.6 

Total 

183 

188.8 

TABLE  11 

Survey  Question  26t  Last  PDR  Risk 

Resolution 

Response 

*  of 

Respondents 

«  of 

Respondents 

Strongly  Agree 

14 

• 

7.7 

Moderately  Agree 

81 

44.3 

Neither  Agree  or  Disagree 

31 

16.9 

Moderately  Disagree 

58 

27.3 

Strongly  Disagree 

7 

3.8 

Total 

183 

188.0 
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TABLE  12 

Survey  Question  27:  Last  PDR  Design  Approach 


•  of 


%  of 


Response 

Respondents 

Respondents 

Strongly  Agree 

22 

12.1 

Moderately  Agree 

90 

49. S 

Neither  Agree  or  Disagree 

33 

18.1 

Moderately  Disagree 

32 

17.6 

Strongly  Disagree 

_ 5 

2.7 

Total 

182 

100.0 

TABLE  13 

Survey  Question  28:  Last  PDR  Technical  Risk 

Response 

#  of 

Respondents 

%  Of 

Respondents 

Strongly  Agree 

8 

4.4 

Moderately  Agree 

60 

33.1 

Neither  Agree  or  Disagree 

55 

30.4 

Moderately  Disagree 

49 

27.1 

Strongly  Disagree 

9 

5.0 

Total 


181 


180.0 


TABLE  14 

Survey  Question  29}  Last  PDR  Interfaces 


Response 

*  of 

Respondents 

«  of 

Respondents 

Strongly  Agree 

11 

6.8 

Moderately  Agree 

84 

46.2 

Neither  Agree  or  Disagree 

33 

18.1 

Moderately  Disagree 

44 

24.2 

Strongly  Disagree 

If 

5.5 

Total 

182 

188.0 

Investigative  Question  lb.  Are  acquiaition  program 
CDRs  conducted  in  accordance  with  the  purposes  stated  in  Mil 
Std  1521? 

Discussion.  Survey  questions  33  through  37  were 
the  stated  purposes  of  a  CDR  outlined  from  Mil  Std  1521.  The 
questions  were  phrased  to  gain  the  repondents  opinion  on 
whether  these  CDR  purposes  were  satisfied  on  the  last  CDR 
the  individual  attended. 

Survey  Question  33: 

The  review  adequately  determined  that  the 
configuration  item  under  review  satisfied  the 
performance  and  engineering  specialty 
requirements  of  the  Cl  development 
specification. 

Survey  Question  34: 

The  review  adequately  determined  that  the 
detailed  design  was  compatible  between  the  Cl 
and  the  other  items  of  equipment,  facilities, 
and  computer  programs. 
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Survey  Question  35 1 


The  review  adequately  assessed  the  Cl  risk 
areas  (on  a  technical,  cost  and  schedule 
basis) . 

Survey  Question  36 t 

The  review  adequately  assessed  the 
producibility  of  the  system  hardware  design. 

Survey  Question  37: 

The  review  adequately  covered  the  preliminary 
product  specification. 

Findings.  Seventy-eight  percent  indicated  moderate 
to  strong  agreement  with  the  stated  purposes  of  survey 
question  33  to  having  been  satisfied  at  their  last  CDR. 

Seventy-two  percent  indicated-  moderate  to  strong 
agreement  with  the  stated  purposes  of  survey  question  34 
having  been  satisfied. 

Sixty  percent  indicated  moderate  to  strong 
agreement  swith  the  stated  purposes  of  survey  question  35 
having  been  satisfied.  Survey  questions  36  and  37  were 
exceptions. 

Only  48  percent  of  all  the  respndents  moderately 
to  strongly  agreed  with  the  stated  purpose  of  survey 
question  36  having  been  satisfied. 

Slightly  over  half,  52.8  percent,  of  all 
respondents  moderately  to  strongly  agreed  with  the  stated 
purpose  of  survey  question  37  having  been  satisfied. 
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TABLE  15 

Survey  Question  33:  Last  CDR  Performance  Requirements 

.  .  '  ■  . 

yii 

Response 

•■■■  *  of 

Respondents 

%  of  . 

Respondents 

Strongly  Agree 

34 

19.1 

Moderately  Agree 

105 

59,0 

l. 

Neither  Agree  or  Disagree 

18 

10.1 

Moderately  Disagree 

19 

10.7 

Strongly  Disagree 

2  ' 

1.1 

Total 

178 

100.0 

Survey  Question  34: 

TABLE  16 

Last  CDR  Design 

Compatibility 

Response 

#  of 

Respondents 

%  of 

Respondents 

Strongly  Agree 

23 

13.0 

Moderately  Agree 

105 

59.3 

Neither  Agree  or  Disagree 

16 

9.0 

Moderately  Disagree 

31 

17.5 

|  Strongly  Disagree 

2 

1.1 

■ 

I  Total 

f 

177 

100.0 

|  TABLE  17 

[  Survey  Question  35:  Last  CDR  Technical  Risk 

i 

Response 

#  of  Respondents 

%  of  Respondents 

Strongly  Agree 

22 

12.4 

Moderately  Agree 

85 

48.0 

Neither  Agree  or  Disagree 

37 

20.9 

Moderately  Disagree 

28 

15,8 

Strongly  Disagree 

5 

2.8 

Total 

177 

100.0 
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TABLE  18 

Survey  Question  36:  Last  CDR  Producibility 


Response 

#  of 

Respondents 

%  of 

Respondents 

Strongly  Agree 

17 

9.7 

Moderately  Agree 

67 

38.3 

Neither  Agree  or  Disagree 

51 

29.1 

Moderately  Disagree 

36 

20.6 

Strongly  Disagree 

4 

2.3 

Total 

175 

100.0 

TABLE  19 

Survey  Question  37:  1 

Last  CDR  Product 

Specification 

*  of 

%  of 

Response 

Respondents 

Respondents 

Strongly  Agree 

17 

9.7 

Moderately  Agree 

76 

43.2 

Neither  Agree  or  Disagree 

41 

23.3 

Moderately  Disagree 

35 

19.9 

Strongly  Disagree 

7 

4.0 

Total 

176 

100.0 

Investigative  Question  2a.  Do  progam  team  members 
agree  with  the  PDR  purposes  outlined  in  Mil  Std  1521? 

Discussion.  Survey  questions  10  through  13  were 
the  stated  purposes  of  a  PDR  in  Mil  Std  1521.  The  questions 
were  phrased  to  get  the  respondents  opinion  as  to  whether 
these  are  PDR  purposes. 
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Survey  Question  10: 

To  evaluate  the  progress,  technical  adequacy, 
and  risk  resolution  (on  a  technical,  cost  and 
schedule  basis)  of  the  selected  design 
approach. 

Survey  Question  11: 

To  determine  the  design  approach 
compatibility  with  performance  and 
engineering  specialty  requirements  of  the 
Configuration  Item  (Cl)  development 
specification. 

Survey  Question  12: 

To  assess  the  technical  risk  associated  with 
the  selected  manufacturing  methods/processes. 

Survey  Question  13: 

To  define  the  physical  and  functional 
interfaces  among  the  Cl  and  other  items  of 
equipment,  facilities,  computer  programs,  and 
personnel. 


Findings.  Eighty-nine  percent  indicated  moderate 
to  strong  agreement  with  the  stated  PDR  purposes  of  the 
survey  question  10. 

Eighty  percent  indicated  moderate  to  strong  agreement 
with  the  stated  PDR  purposes  of  the  survey  question  11. 

Fifty-three  percent  indicated  moderate  to  strong 
agreement  with  the  stated  PDR  purposes  of  the  survey 
question  12. 

Sixty-five  percent  indicated  moderate  to  strong 
agreement  with  the  stated  PDR  purposes  of  the  survey 
question  13. 
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TABLE  20 

Survey  Question  10:  PDR  Technical  Adequacy 


Response 

#  of 

Respondents 

%  of 

Respondents 

Strongly  Agree 

105 

42.2 

Moderately  Agree 

118 

47.4 

Neither  Agree  or  Disagree 

12 

4.8 

Moderately  Disagree 

12 

4.8 

Strongly  Disagree 

2 

0.8 

Total 

249 

100.0 

Survey  Question 

TABLE  21 

11:  PDR  Design 

Approach 

Response 

#  of 

Respondents 

%  of 

Respondents 

Strongly  Agree 

86 

34.4 

Moderately  Agree 

115 

46.0 

Neither  Agree  or  Disagree 

35 

14.0 

Moderately  Disagree 

14 

5.6 

Total 

250 

100.0 

Survey  Question 

TABLE  22 

12:  PDR  Technical  Risk 

Response 

#  of 

Respondents 

%  of 

Respondents 

Strongly  Agree 

35 

14.0 

Moderately  Agree 

99 

39.6 

Neither  Agree  or  Disagree 

57 

22.8 

Moderately  Disagree 

47 

18.8 

Strongly  Disagree 

12 

4.8 

Total 

250 

100.0 
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TABLE  23 

Survey  Question  13:  PDR  interfaces 


Response 

I  of 

Respondents 

t  of 

Respondents 

Strongly  Lgree 

57 

23.1 

Moderately  Agree 

104 

42.1 

Neither  Agree  or  Disagree 

42 

17.0 

Moderately  Disagree 

34 

13.8 

Strongly  Disagree 

10 

4.0 

Total 

247 

100.0 

Investigative  Question 

2b.  Do  program 

team  members 

agree  with  the  CDR  purposes  outlined  in  Mil  Std  1521? 

Discussion.  Survey  questions  17  through  22  were 
the  stated  purposes  of  a  CDR  in  Nil  Std  1521.  The  questions 
were  phrased  to  get  the  respondents  opinion  as  to  whether 
these  are  CDR  purposes. 

Survey  Question  17: 

To  determine  that  the  detailed  design  of  the 
configuration  item  under  review  satisfies  the 
performance  and  engineering  specialty 
requirements  of  the  Cl  development 
speci f icat ions . 

Survey  Question  18: 

To  establish  the  detail  design  compatibility 
among  the  Cl  and  other  items  of  equipment, 
facilities,  computer  programs. 

Survey  Question  19: 

To  assess  the  configuration  item  risk  areas 
(on  a  technical,  cost  and  schedule  basis). 


I 
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Survy  Question  28: 

To  assets  the  results  of  the  producibility 

analyses  conducted  on  system  hardware  design. 

Survey  Question  21 t 

To  review  the  preliminary  product 

specification. 

Survey  Question  22: 

To  review  major  design  modifications. 

Findings.  The  trend  for  the  respondents  was  96 , 
90.4,  78,  66.9,  54.8  and  59.8  percent  for  survey  questions 
17,  18,  19,  20,  21,  and  22  respectively,  indicating  moderate 
to  strong  agreement  with  the  stated  CDR  purposes  of  the 
survey  questions. 


TABLE  24 

Survey  Question  17:  CDR  Performance  Requirements 

#  of  %  of 

Response _  Respondents  Respondents 


Strongly  Agree 

150 

60.0 

Moderately  Agree 

90 

36.0 

Neither  Agree  or  Disagree 

5 

2.0 

Moderately  Disagree 

5 

2.0 

Total 

250 

100.0 

47 


TABLE  25 

Survey  Question  18:  CDR  Design  Compatibility 


Response 

*  of 

Respondents 

%  of 

Respondents 

Strongly  Agree 

105 

42.0 

Moderately  Agree 

121 

48.4 

Neither  Agree  or  Disagree 

10 

4.0 

Moderately  Disagree 

12 

4.8 

Strongly  Disagree 

2 

0.8 

Total 

250 

100.0 

Survey  Question 

TABLE  26 

19:  CDR  Risk  Resolution 

Response 

#  of 

Respondents 

%  of 

Respondents 

Strongly  Agree 

57 

22.8 

Moderately  Agree 

138 

55.2 

Neither  Agree  or  Disagree 

31 

12.4 

Moderately  Disagree 

20 

8.0 

Strongly  Disagree 

4 

1.6 

Total 

250 

100.0 
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TABLE  27 

Survey  Question  20:  CDR  Producibil ity 


Response 

*  of 

Respondents 

%  of 

Respondents 

Strongly  Agree 

41 

16.7 

Moderately  Agree 

123 

50.2 

Neither  Agree  or  Disagree 

57 

23.2 

Moderately  Disagree 

21 

8.6 

Strongly  Disagree 

3 

1.2 

Total 

245 

100.0 

Survey  Question  21 

TABLE  28 

:  CDR  Product  Specification 

Response 

*  of 

Respondents 

%  Of 

Respondents 

Strongly  Agree 

33 

13.3 

Moderately  Agree 

103 

41.5 

Neither  Agree  or  Disagree 

49 

19.8 

Moderately  Disagree 

48 

19.4 

Strongly  Disagree 

15 

6.0 

Total 

248 

100.0 

TABLE  29 

Survey  Question  22:  CDR  Design  Modifications 
Response _  ♦  of  Respondents  %  of  Respondents 


Strongly  Agree 

63 

25.3 

Moderately  Agree 

86 

34.5 

Neither  Agree  or  Disagree 

29 

11.6 

Moderately  Disagree 

55 

22.1 

Strongly  Disagree 

16 

6.4 

Total 

249 

100.0 
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Investigative  Question  3a.  Do  experienced  PDR 
individuals  perceive  the  Mil  Std  1521  PDR  purposes  to  be  of 
greater  importance  than  do  those  without  PDR  experience? 


Discussion.  Survey  questions  6  and  10  through  13 
was  used  to  address  this  investigative  question.  This 
question  addressed  how  useful  the  Mil  Std  is  to  managers  and 
engineers  by  examining  the  separate  opinions  of  those 
respondents  that  have  and  have  not  participated  in  a  PDR. 

The  cross  tabulation  was  used  to  examine  the 
relationship  of  the  responses  to  survey  questions  10  through 
13  against  the  PDR  experience  variable. 

Survey  Question  6: 

Number  of  Preliminary  Design  Reviews  you  have 
formally  participated  in. 

Survey  Question  10: 

To  evaluate  the  progress ,  technical  adequacy, 
and  risk  resolution  (on  a  technical,  cost  and 
schedule  basis)  of  the  selected  design 
approach. 

Survey  Question  11: 

To  determine  the  design  approach 
compatibility  with  performance  and 
engineering  specialty  requirements  of  the 
Configuration  Item  (Cl)  development 
specification. 

Survey  Question  12; 


To  assess  the  technical  risk  associated  with 
the  selected  manufacturing  methods/processes. 


Survy  Question  13 


To  define  the  physical  and  functional 
interfaces  among  tha  CX  and  other  items  of 
equipment,  facilities,  coaputar  programs,  and 
personnel. 


Findings*  Tha  croas  tabulation  of  tha  vaciablas 
cavaalad  a  trand  of  aodatataly  agraa  for  all  four  purposas 
addressed  by  tha  survay  quastion  If  -  13. 


TABLB  30 

FOR  Participation  vs  Adequacy  of  Design  Approach 

Had  Particpated  in  a  PDR 


Response 

YES 

NO 

Strongly  Agree 

44.51 

34.37 

Moderately  Agree 

47.25 

48.44 

Neither  Agree  or  Disagree 

2 .20 

12.50 

Moderately  Disagree 

5.49 

3.13 

Strongly  Disagree 

0.55 

1.56 

TABLE  31 

PDR  Participation  vs  Design  Approach  Compatibility  with 

Development  Specification 

Response 

Had  Particpated 
YES 

in  a  PDR 

NO 

Strongly  Agree 

35.16 

31.25 

Moderately  Agree 

45.60 

46.87 

Neither  Agree  or  Disagree 

13.74 

15.62 

Moderately  Disagree 

5.49 

6.25 
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TABLE  32 

PDR  Participation  va  Manufacturing  Technical  Risk 

Had  Partiepatad  in  a  PDR 


Response 

YES 

NO 

Strongly  Agree 

14.29 

12.50 

Moderately  Agree 

36.81 

48.44 

Neither  Agree  or  Disagree 

21.43 

26.56 

Moderately  Disagree 

21.98 

9.37 

Strongly  Disagree 

5.49 

3.13 

TABLE  33 

PDR  Participation 

vs  Interface 

Definition 

Had  Particpated  in  a  PDR 

Response 

YES 

NO 

Strongly  Agree 

24.02 

18.75 

Moderately  Agree 

43.02 

40.62 

Neither  Agree  or  Disagree 

15.08 

23.44 

Moderately  Disagree 

13.41 

14.06 

Strongly  Disagree 

4.47 

3.13 

Investigative  Question 

3b.  Do  CDR 

experienced 

individuals  perceive  the  Mil  Std  1521  CDR  purposes  to  be  of 
greater  importance  than  do  those  without  CDR  experience? 

Discussion.  Survey  questions  6  and  17  through  22 
was  used  to  address  this  investigative  question.  This 
question  addressed  how  useful  the  Mil  Std  is  to  managers  and 
engineers  by  examining  the  separate  opinions  of  those 
respondents  that  have  and  have  not  participated  in  a  CDR. 
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The  cross  tabulation  was  used  to  examine  tha 
calationahip  of  tha  taaponaaa  to  survey  quaationa  1?  through 
22  againat  tha  CDR  axparianca  variabla. 

Survay  Quaatlon  6: 

Number  of  Preliminary  oaaign  Reviews  you  hava 
formally  participated  in. 

Survay  Quaation  17 t 

To  datarmina  that  tha  datailad  design  of  tha 
configuration  item  under  review  satisfies  tha 
performance  and  engineering  specialty 
requirements  of  tha  Cl  development 
specifications. 

Survey  Question  18s 

To  establish  the  detail  design  compatibility 
among  the  Cl  and  other  items  of  equipment, 
facilities,  computer  programs. 

Survey  Question  19: 

To  assess  the  configuration  item  risk  areas 
(on  a  technical,  cost  and  schedule  basis). 

Survey  Question  28: 

To  assess  the  results  of  the  producibility 
analyses  conducted  on  system  hardware  design. 

Survey  Question  21: 

To  review  the  preliminary  product 
specification. 

Survey  Question  22: 

To  review  major  design  modifications. 


Findings.  The  cross  tabulation  of  the  variables 
revealed  a  trend  of  moderate  to  strong  agreement  for  all 
four  purposes  addressed  by  the  survey  question  17  -  22.  For 
survey  question  21  the  responses  were  independent  of  whether 


the  individual  had  or  had  not  participated  in  a  CDR.  Tha 
distribution  between  thosa  that  agraa  and  these  that 
disagree  warn  approximately  tha  same  with  a  slight  tandancy 
toward  moderate  agreement,  but  a  total  of  45.59  parcant  and 
36.93  parcant  of  thosa  who  hava  not  and  have,  respectively, 
attended  COR  had  a  tandancy  to  neither  agraa  nor  disagraa  or 
Hodaratadly  disagraa  with  tha  stated  purposes  of  this  survey 
question* 


TABLE  34 

CDR  Participation  vs  Adequacy  of  Design  Approach 
Compatibility  with  tha  Development  Specification 


Had  Particpated  in  u  CDR 


Response 

YES 

NO 

Strongly  Agree 

64.61 

48.53 

Moderately  Agree 

33.15 

42.65 

Neither  Agree  or  Disagree 

.56 

5.88 

Moderately  Disagree 

1.69 

2.94 

TABLE  35 

CDR  Participation  vs  Establish  Interface  Definition 

Had  Particpated  in  a  CDR 


Response 

YES 

NO 

Strongly  Agree 

43.82 

36.76 

Moderately  Agree 

49.44 

45.59 

Neither  Agree  or  Disagree 

1.69 

10.29 

Moderately  Disagree 

4.49 

5.88 

Strongly  Disagree 


0.56 


1.47 


TABLE  36 

COR  Participation  vs  Asaass  Risk  Areas 


Had  Partiepatad  in  a  CDR 


Response 

YES 

NO 

Strongly  Agree 

25.28 

16.18 

Moderately  Agree 

53.37 

60.29 

Neither  Agree  or  Disagree 

IE. 67 

16.18 

Moderately  Disagree 

9.55 

4.41 

Strongly  Disagree 

1.12 

2.94 

TABLE  37 

CDR  Participation  vs  Assess  Producibi iity 

Had  Partiepatad  in  a  CDR 


Response 

YES 

NO 

Strongly  Agree 

17.34 

16.18 

Moderately  Agree 

50.29 

52.94 

Neither  Agree  or  Disagree 

24.28 

20.59 

Moderately  Disagree 

6.94 

8.82 

Strongly  Disagree 

1.16 

1.47 

TABLE  38 

CDR  Participation  vs  Review 
preliminary  Product  Specification 

Had  Partiepated  in  a  CDR 


Response 

YES 

NO 

Strongly  Agree 

15.34 

8.82 

Moderately  Agree 

41.48 

41.18 

Neither  Agree  or  Disagree 

21.02 

17.65 

Moderately  Disagree 

15.91 

27.94 

Strongly  Disagree 

6.25 

4.41 
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TABLE  39 

CDR  Participation  vs  Review 
Major  Design  Modifications 

Had  Particpated  in  a  CDR 


Response 

YES 

NO 

Strongly  Agree 

27.68 

20.59 

Moderately  Agree 

34.46 

35.29 

Neither  Agree  or  Disagree 

10.17 

14.71 

Moderately  Disagree 

20.34 

25.00 

Strongly  Disagree 

7.34 

4.41 

PDR/CDR  Ef f ect iveness/Ef f ici 

ency  Measures 

Tables  40  through  63  covers  the  category  of  PDR  and  CDR 
effectiveness  and  efficiency  measures. 

Investigative  questions  four  through  sixteen  posed  in 
chapter  one  are  addressed  here. 

Investivat ive  Question  4a.  What  approval  was  given  at 
the  conclusion  of  the  PDR? 

Discussion .  Survey  question  32  asked  what  was  the 
PDR  approval  rating  given  at  the  last  PDR  participated  in  by 
the  individual  respondents. 

Findings .  Sixty-four  percent  of  the  respondents 
indicated  the  last  PDR  they  participated  in  was  approved 
contingent  upon  some  action  to  be  completed.  Twenty-one 
percent  indicated  the  PDR  was  approved  outright  and  10.5 
percent  did  not  know  what  approval  rating  was  given.  Only 
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0.6  percent  indicated  the  PDR  last  participated  in  was 
disapproved. 


TABLE  40 

Survey  Question  32:  PDR  Approval 


Response 

#  of 

Respondents 

%  of 

Respondents 

Approved 

39 

21.5 

Approved  Contingent 

117 

64.6 

Approved  with  waiver 

5 

2.8 

Disapproved 

1 

0.6 

Did  Not  Know 

19 

10.5 

Total 

181 

100.0 

Investigative  Question  4b.  What  approval  was  given  at 
the  conclusion  of  the  CDR? 

Discussion.  Survey  queston  41  asked  what  was  the 
CDR  approval  rating  given  at  the  last  CDR  participated  in  by 
the  individual  respondents. 

Findings .  Sixty-three  percent  of  the  respondents 
indicated  the  last  CDR  they  participated  in  was  approved 
contingent  upon  some  action  to  be  completed.  Twenty-one 
percent  indicated  the  CDR  was  approved  outright  and  8.5 
percent  did  not  know  what  approval  rating  was  given.  Only 
2.8  percent  indicated  the  last  CDR  they  participated  in  was 
disapproved. 
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TABLE  41 

Survey  Question  41:  CDR  Approval 


Response 

#  of 

Respondents 

%  of 

Respondents 

Approved 

37 

21.0 

Approved  Contingent 

112 

63.6 

Approved  with  waiver 

7 

4.0 

Disapproved 

5 

2.8 

Did  Not  Know 

15 

8.5 

Total 

176 

100.0 

Investigative  Question  5a.  Did  the  PDR  provide  an 
adequate  evaluation  o£  the  system  to  proceed  into  the  next 
acquisition  phase? 

Discussion.  Survey  question  6  and  56  were 
analyzed  together  to  determine  the  respondents  opinions  of 
how  adequate  the  PDR  was  in  evaluating  the  system. 

Survey  Question  6: 

Number  of  Preliminary  Design  Reviews  you  have 
formally  participated  in. 

Survey  Question  56: 

In  your  opinion,  did  the  last  or  current 
program  PDR  and  CDR  you  attended  adequately 
evaluate  the  system  to  allow  it  to  proceed 
into  the  next  acquisition  phase. 

Findings .  Seventy  percent  of  the  respondents 
indicated  the  last  PDR  they  had  participated  in  adequately 
evaluated  the  system. 

Examination  of  responses  to  survey  question  56  with  the 
responses  grouped  into  those  with  PDR  experience  and  those 


without  FDR  experience  the  result  were  different.  Of  those 
individuals  having  participated  in  PDRs,  77.22  percent 
indicated  the  last  FDR  adequately  evaluated  the  system.  Of 
those  that  had  no  previous  PDR  experience  only  26.67  percent 
believed  the  PDR  adequately  evaluated  the  system,  and  70 
percent  did  not  know  whether  the  review  was  adequate  or  not. 


TABLE  42 

PDR  Participation  vs  Adequacy  of 
System  Evaluation 


Response 

Had  Participated 
YES 

in  a  PDR 
NO 

Yes 

77.22 

26.67 

No 

18.33 

3.33 

Did  Not  Know 

3.33 

70.00 

Not  Important 

1.11 

— 

Investigative  Question 

5b.  Did  the  CDR 

provide  an 

adequate  evaluation  of  the  system  to  proceed  into  the  next 
acquisition  phase? 

Discussion .  Survey  question  6  and  56  were 
analyzed  together  to  determine  the  respondents  opinions  of 
how  adequate  the  PDR  was  in  evaluating  the  system. 

Survey  Question  6: 

Number  of  Preliminary  Design  Reviews  you  have 
formally  participated  in. 

Survey  Question  56: 

In  your  opinion,  did  the  last  or  current 
program  PDR  and  CDR  you  attended  adequately 
evaluate  the  system  to  allow  it  to  proceed 
into  the  next  acquisition  phase. 


Findings.  Sixty-one  percent  of  the  respondents 
indicated  the  last  CDR  they  had  participated  in  adequately 
evaluated  the  system. 

Examination  of  responses  to  survey  question  56  with  the 
responses  grouped  into  those  with  CDR  experience  and  those 
without  CDR  experience  the  result  are  much  different,  of 
those  individuals  having  participated  in  CDR  69.49  percent 
indicated  the  last  CDR  adequately  evaluated  the  system.  Of 
those  that  had  no  previous  CDR  experience,  only  26.32 
percent  believed  the  PDR  adequately  evaluated  the  system, 
and  65.79  percent  did  not  know  whether  the  review  was 
adequate  or  not. 


TABLE  43 

CDR  Participation  vs  Adequacy  of 
System  Evaluation 

Response 

Had  Participated  in 
YES 

a  CDR 
NO 

Yes 

69.49 

26.32 

No 

23.73 

5.26 

Did  Not  Know 

5.65 

65.79 

Not  Important 

1.13 

2.63 

Investigative  Question  6a.  Were  all  AIs  resolved 
before  the  PDR  approval  was  given? 

Discussion.  Survey  question  31  asked  if  all 
action  items  were  resolved  before  an  approval  of  the  PDR  was 
given. 


Findings .  Forty-eight  percent  of  the  respondents 
indicated  all  AIs  were  not  resolved  before  the  PDR  approval 


was  given.  Thirty-three  percent  of  the  respondents 
indicated  all  AIs  were  resolved  before  the  PDR  approval  was 
given  and  18.2  percent  did  not  know  if  the  AIs  were  resolved 
before  the  PDR  approval  was  given. 


TABLE  44 

Survey  Question  31:  PDR  Action  Item  Resolution 


Response 

I  of 

Respondents 

%  of 

Respondents 

Yes 

60 

33.1 

No 

88 

48.6 

Did  Not  Know 

33 

18.2 

Total 

181 

99.9 

Investigative 

Question  6b.  Were  all 

AIs  resolved 

before  the  CDR  approval  was  given? 

Discussion.  Survey  question  39  asked  if  all 


action  items  were  resolved  before  an  approval  of  the  CDR  was 
given. 

Findings .  Forty-four  percent  of  the  respondents 
indicated  all  AIs  were  not  resolved  before  the  CDR  approval 
was  given.  Thirty-nine  percent  of  the  respondents  indicated 
all  AIs  were  resolved  before  the  CDR  approval  was  given  and 
15  percent  did  not  know  if  the  AIs  were  resolved  before  the 
CDR  approval  was  given. 


61 


Survey  Question 

TABLE  45 

39:  CDR  Action  Item 

Resolution 

Response 

*  of 

Respondents 

4  of 

Respondents 

Yes 

67 

39.9 

No 

77 

44.5 

Did  Not  Know 

26 

15.0 

Not  Important 

1 

0.6 

Total 

171 

100.0 

Investigative  Question  7a.  Were  any  Als  left  open  at 
the  conclusion  of  the  PDR? 

Discussion.  Survey  question  45  asked  if  any  Als 
were  left  opened  at  the  conclusion  of  the  PDR. 

Findings.  Seventy-three  percent  of  the 
respondents  indicated  there  were  action  items  left  open  at 
the  conclusion  of  the  PDR  they  participated  in  last. 

Seventeen  percent  of  the  respondents  indicated  they  did 
not  know  if  any  action  items  were  left  open  at  the 
conclusion  of  the  PDR. 

TABLE  46 

Survey  Question  45:  PDR  Open  Action  Items 


Response 

#  of 

Respondents 

4  of 

Respondents 

Yes 

156 

73.9 

No 

18 

8.5 

Did  Not  Know 

37 

17.5 

Total 

211 

99.9 
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Investigative  Question  7b.  Were  any  AIs  left  open  at 
the  conclusion  of  the  CDR? 

Discussion.  Survey  question  4S  asked  if  any  AIs 
were  left  opened  at  the  conclusion  of  the  CDR. 

Findings.  Sixty-five  percent  of  the  respondents 
indicated  there  were  action  items  left  open  at  the 
conclusion  of  the  CDR  they  participated  in  last. 

Twenty-three  percent  of  the  respondents  indicated  they 
did  not  know  if  any  action  items  were  left  open  at  the 
conclusion  of  the  CDR. 

TABLE  47 

Survey  Question  45:  CDR  Open  Action  Items 


Response 

I  of 

Respondents 

%  of 

Respondents 

Yes 

148 

65.8 

NO 

23 

10.2 

Did  Not  Know 

53 

23.6 

Not  important 

1 

0.4 

Total 

225 

100.0 

Investigative  Question 

8a.  Of  AIs 

presented  at  the 

were  any  issued  previously  and  were  these  closed  prior  to 
the  PDR  completion? 


Discussion.  Survey  question  46  asked  if  AIs 
presented  at  the  PDR  already  existed  as  previous  meeting 
issues.  Survey  questions  47  asked  if  these  action  items 
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we re  resolved  prior  to  the  conpletion  of  the  PDR.  Only  the 
reponses  of  individuals  having  participated  in  a  PDR  were 
examined. 

Findings.  Over  79.13  percent  of  the  respondents 
indicated  the  AI  presented  at  the  PDR  already  existed.  Only 
42.86  percent  claimed  that  existing  AIs  were  resolved  or 
closed  prior  to  the  completion  of  the  PDR.  Over  half,  51.65 
percent  indicated  these  AIs  were  not  resolved  or  closed 
prior  to  the  completion  of  the  PDR. 

Investigative  Question  8b.  Of  AIs  presented  at  the  CDR 
how  many  were  issued  previously  and  were  these  closed  prior 
to  the  CDR  completion? 

Discussion.  Survey  question  46  asked  if  AIs 
presented  at  the  CDR  already  existed  as  previous  meeting 
issues.  Survey  questions  47  asked  if  these  action  items 
were  resolved  prior  to  the  completion  of  the  CDR.  Only  the 
responses  of  individuals  having  par iticipated  in  a  CDR  were 
examined. 

Findings .  Over  86.78  percent  of  the  respondents 
indicated  that  the  AIs  presented  at  the  CDR  already  existed. 
Only  39.05  percent  claimed  that  existing  AIs  were  resolved 
or  closed  prior  to  the  completion  of  the  PDR.  Fifty-seven 
percent  indicated  these  AIs  were  not  resolved  or  closed 
prior  to  the  completion  of  the  CDR. 


64 


investigation  Question  9a.  Should  all  Ala  ba  resolved 
before  entering  a  PDR? 

Discussion.  Survey  question  6,  9  and  15  response 
data  were  compiled  and  analysed  against  this  investigative 
question.  The  survey  question  asked  if  the  respondent 
believed  that  all  AIs  should  be  resolved  before  entering  a 
PDR.  Survey  question  6  and  9  provided  the  data  to  determine 
if  the  responses  were  dependent  upon  their  PDR  experience 
and  DAFSC *  respectively. 

findings.  Of  those  individuals  having 
participated  in  PDRs,  50  percent  of  DAFSC  2716  claimed  all 
action  items  should  be  resolved  before  approving  the  PDR, 
58.62  percent  of  DAFSC  2816  claimed  all  action  items  should 
be  resolved  before  approving  the  PDR,  52.94  percent  of  DAFSC 
2724  claimed  all  action  items  should  be  resolved  before 
approving  the  PDR  and  49.38  percent  of  DAFSC  28X5  claimed 
all  action  items  should  be  resolved  before  approving  the 
PDR. 


TABLE  48 

DAFSC  vs  Prior  To  PDR 
Action  Items  Resolution 


DAFSC 

YES 

NO 

DO  NOT 
KNOW 

NOT 

IMPORTANT 

2716 

50.00 

41.67 

5.56 

2.78 

2816 

34.48 

58.62 

6.90 

— 

2724 

52.94 

41.18 

5.88 

— 

28X5 

43.21 

49.38 

6.17 

1.23 
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Investigation  Question  9b.  Should  all  Ala  be  resolved 


before  entering  a  CDR? 

Discussion.  Survey  question  6,  9  and  23  response 
data  were  compiled  and  analysed  against  this  investigative 
question.  The  survey  question  asked  if  the  respondent 
believed  that  all  AIs  should  be  resolved  before  entering  a 
CDR.  Survey  question  6  and  9  provided  the  data  to  determine 
if  the  responses  were  dependent  upon  their  CDR  experience 
and  DAFSC,  respectively. 

Findings.  Of  those  individuals  having 
participated  in  CDRs ,  54.55  percent  of  DAFSC  2716  claimed 
all  action  items  should  be  resolved  before  approving  the 
CDR,  62.07  percent  of  DAFSC  2816  claimed  all  action  items 
should  be  resolved  before  approving  the  CDR,  60.61  percent 
of  DAFSC  2724  claimed  all  action  items  should  be  resolved 
before  approving  the  CDR  and  78.31  percent  of  DAFSC  28X5 
claimed  all  action  items  should  be  resolved  before  approving 
the  CDR. 

TABLE  49 

DAFSC  vs  Prior  To  CDR 
Action  Items  Resolution 


DAFSC 

YES 

NO 

DO  NOT 
KNOW 

NOT 

IMPORTANT 

2716 

54.55 

39.39 

3.03 

3.03 

2816 

62.07 

34.48 

3.45 

— 

2724 

60.61 

30.30 

6.06 

— 

28X5 

78.31 

18.07 

1.20 

2.41 
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Investigative  Question  It.  We re  there  slippages  in  the 
systea/subsystem  design/development  schedule  and  CDR 
schedule? 

Discussion.  Survey  question  44  asked  if  the  last 
prograa  CDR  participated  in  had  a  CDR  schedule  slip  and  the 
nuaber  of  weeks  slipped.  Survey  question  52  asked  if  the 
last  prograa  worked  if  there  were  slips  in  the  systea  and 
subsystem  design  and  development  and  nuaber  of  subsystems 
and  weeks  slipped. 

Bach  question  results  are  presented  and  the  results  of 
crosstabulation  and  the  chi  square  test  for  dependency  on 
these  two  questions  was  used  to  reveal  the  relationship  of 
the  slippage  in  system  design  and  CDR  slippage. 

findings.  Sixty  percent  of  all  respondents 
indicated  the  system/subsystem  design/development  schedule 
slipped  on  their  last  program.  Sixteen  percent  indicated  no 
slippage  and  22.2  percent  did  not  know. 

TABLE  50 

Survey  Question  52:  System  Development 
Schedule  Slip 

#  of  %  of 

Response _  Respondents  Respondents 


Yes 

136 

60.4 

No 

37 

16.4 

Did  Not  Know 

50 

22.2 

Not  Important 

2 

0.9 

Total 

225 

99.9 
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Fifty-nine 

percent  of  all  respondents 

indicated  the  CDR 

slipped  on  their 

last  program.  Seventeen  percent  indicated 

no  slippage  and  21.4  percent  did  not  know. 

TABLE  51 

Survey  Question  44:  CDR  Schedule  Slip 

Response 

«  of 

Respondents 

1  of 

Respondents 

Yes 

138 

59.0 

No 

42 

17.9 

Did  Not  Know 

50 

21.4 

Not  important 

3 

1.3 

Total 

233 

99.6 

Analysis  of  the  results  of  the  two  survey  questions 
revealed  that  74.07  percent  of  those  respondents  having  said 
yes  to  system  design/development  slippage  also  indicated  yes 
to  the  CDR  slippage.  Of  those  respondents  indicating  no  to 
system  design/development  slippage  55.56  percent  indicated 
no  to  CDR  slippage.  The  chi  square  goodness  of  fit  test 
revealed  strong  dependency  between  the  responses  of  the  two 
questions . 

The  largest  group  of  respondents  indicated  their  last 
system/subsystem  design/development  and  CDR  schedules 
slipped  (45.05  percent  o'  all  respondents). 
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TABLE  52 

CDR  Slip  vs  Syatem/Subsystem  Slip 


percent 
row  percent 
col  percent 

CDR 

Slip 

YES 

System/Subsystem  Slip 

DO  NOT  NOT 

NO  KNOW  IMPORTANT 

YBS 

45.05 

5.86 

8.56 

0.45 

75.19 

9.77 

14.29 

0.75 

74.07 

36.11 

38.78 

50.00 

NO 

5.86 

9.01 

3.15 

0.45 

31.71 

48.78 

17.07 

2.44 

9.63 

55.56 

14.29 

50.00 

DO  NOT 

8.56 

1.35 

9.91 

KNOW 

43.18 

48.78 

17.07 

2.44 

14.07 

8.33 

44.90 

-  - 

NOT 

0.90 

_  _ 

0.45 

_  _ 

IMPORTANT 

66.67 

— 

33.33 

— 

1.48 

— 

2.04 

-- 

STATISTIC _ DEGREE  OF  FREEDOM  VALUE  PROBABILITY 

CHI-SQUARE  12  67.610  0.000 


The  data  revealed,  of  those  respondents  having 
indicated  slips  in  the  system/subsystem  design/development, 
there  were  97  system/subsytems  slipped  for  an  average  of 
over  2  per  program  with  a  standard  deviation  of  2.30  and  a 
range  from  0  to  12. 

The  average  length  of  a  slip  was  17.3  weeks  with  a 
standard  deviation  of  18.51  and  a  range  from  0  to  99  weeks. 

Of  the  126  respondent  claiming  a  slip  in  the  CDR 
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schedule  the  average  slip  was  10.48  weeks  with  a  standard 
deviation  of  11.13  and  a  range  from  1  to  56  weeks. 

Investigative  question  11.  Should  the  design  be 
complete  before  approving  a  CDR? 

Discussion.  Survey  question  6,9  and  42  were  used 
to  ana*’"?e  this  question.  Only  the  individuals  with  previous 
CDR  experience  were  considered  against  the  different  DAFSCs. 

Survey  Question  6t 

Number  of  Preliminary  Design  Reviews  you  have 
formally  participated  in. 

Survey  Question  9: 

Duty  AFSC. 

Survey  Question  42: 

The  CDR  should  not  be  conducted  until  the 
detailed  design  of  each  configuration  item  is 
complete. 

Findings .  Of  the  respondents  having  participated 
in  a  CDR,  68.75  percent  of  the  DAFSC  2716  said  the  CDS 
should  not  be  conducted  until  the  detailed  design  of  each 
configuration  item  is  complete,  64.29  percent  of  DAFSC  2816, 
67.74  percent  of  DAFSC  2724  and  75.90  percent  of  DAFSC  28X5 
agreed  that  a  design  should  be  complete  before  conducting  a 
CDR. 
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TABLE  53 

OAFSC  vs  Survey  Question  42:  Complete  Design 


DAFSC 

YES 

NO 

DO  NOT 
KNOW 

NOT 

IMPORTANT 

2716 

68.75 

31.25 

— 

— 

2816 

64.29 

35.71 

— 

-- 

2724 

67.74 

25.81 

6.45 

-- 

28X5 

75.90 

19.28 

4.82 

— 

Investigative  question  12. 

Were  major 

design  problems 

and  modifications  presented  at  the  CDR  and  did  any  of  these 
exist  previously? 

Discussion.  Survey  question  40  asked  if  there 
were  any  major  design  modifications  presented-at  the 
respondents  last  CDR  and  if  there  were,  how  many.  Survey 
question  48  asked  if  design  problems  presented  at  the 
respondents  last  CDR  were  covered  in  previous  meetings. 

Findings .  Forty-six  percent  of  the  respondents 
claimed  major  design  modifications  were  presented  at  their 
last  CDR.  Thirty-nine  percent  said  none  were  presented  and 
13.1  percent  did  not  know. 


TABLE  54 

Survey  Question  40:  CDR  Design  Modifications 


Response 

#  of 

Respondents 

%  of 

Respondents 

Yes 

81 

46.0 

No 

70 

39.8 

Did  Not  Know 

23 

13.1 

Not  Important 

2 

1.1 

Total 

176 

100.0 

The  quantity  of  design  mods  present  averaged  3.22  per 
positve  response  to  this  question.  This  had  a  standard 
deviation  of  2.595  and  the  number  of  mods  presented  ranged 
from  0  to  15. 

Sixty-nine  percent  of  the  respondents  indicated  the 
design  problems  presented  at  their  last  CDR  were  covered  in 
previous  meetings.  Only  6.9  percent  said  no  and  22.8  percent 
did  not  know. 


TABLE  55 

Survey  Question  48:  Previous  Coverage 
of  CDR  Design  Modifications 


Response 

#  of 

Respondents 

%  of 

Respondent 

Yes 

162 

69.8 

No 

16 

6.9 

Did  Not  Know 

53 

22.8 

Not  Important 

1 

0.4 

Total 

232 

99.9 
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Investigative  question  13.  Were  the  CDR  supporting 
data  packages  effective  in  supporting  the  system  review 
(i.e.  complete,  delivered  on  time)? 

Discussion.  Survey  question  50  asked  if  the 
respondent  had  adequate  time  to  review  the  CDR  supporting 
data  packages  to  their  satisfaction  and  were  their  comments 
addressed  sufficiently  by  the  contractor  prior  to  the  CDR. 

Survey  question  51  asked  if  the  respondent  last  CDR 
supporting  data  packages  were  complete  and  delivered  on 
schedule. 

Findings .  Forty-nine  percent  of  the  respondents 
claim  they  did  not  have  adequate  time  to  review  the  CDR 
supporting  data  packages  to  their  satisfaction  and  that  the 
comments  were  insufficiently  addressed  by  the  contractor 
prior  to  the  CDR.  Thirty-six  percent  indicated  they  did  have 
adequate  time  and  comments  were  addressed  by  the  contractor 
sufficiently,  and  13.8  percent  did  not  know. 

TABLE  56 

Survey  Question  50:  Adequate  Review  Time 
for  CDR  Data 

#  of  %  of 

Response _  Respondents  Respondents 


Yes 

81 

36.0 

No 

112 

49.8 

Did  Not  Know 

31 

13  8 

Not  Important 

1 

0.4 

Total 

225 

100.0 
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Only,  44.1  percent  of  the  respondents  indicated  the  CDR 


packages  were  complete.  While  37.6  percent 

indicated  the  CDR 

data  packages  were  incomplete  and  17.8  percent  did  not  know. 

TABLE  57 

CDR  Data  Package  Complete 

Survey  Question  51 

Response 

#  of 

Respondents 

%  of 

Respondents 

Yes 

89 

44.1 

NO 

76 

37.6 

Did  Not  Know 

36 

'J 

• 

CO 

Not  Important 

1 

0.5 

Total 

202 

100.0 

Thirty-six  percent  indicated  the  CDR 

data  packages  were 

delivered  on  schedule.  Forty-four  percent 

indicated  they 

were  not  and  18.1  percent 

did  not  know. 

CDR 

TABLE  58 

Data  Package  Delivered  On 
Survey  Question  51 

Time 

Response 

#  of 

Respondents 

%  of 

Respondents 

Yes 

80 

36.2 

No 

99 

44.8 

Did  Not  Know 

40 

18.1 

Not  Important 

2 

0.9 

Total 

221 

100.0 

Investigative  question  14a.  Should  there  be  a  system 
overview  or  tutorial  as  part  of  a  PDR  and  how  much  of  a  PDR 
should  be  devoted  to  system  overview  and  tutorials? 

Discussion.  Survey  question  16  asked  how  many 
hours  of  a  PDR  should  be  devoted  to  system  overview  and 
tutorial  information.  Survey  questions  6  and  9  were  used  to 
gather  the  responses  to  question  16  by  the  group  of 
individuals  having  participated  in  a  PDR  and  identifing 
responses  against  the  DAFSC. 

Findings.  Of  the  74  percent  of  the  respondents 
having  participated  in  PDRs;  65.17  (Table  4.59)  percent 
indicated  less  than  2  hours  of  a  PDR  should  be  devoted  to 
system  overview  and  tutorial,  ‘29.44  percent  claimed  3  to  5 
hours,  3.89  percent  claimed  5  to  9  hours  and  1.11  percent 
claimed  10  or  more  hours. 

Fifty-eight  percent  of  DAFSC  2716,  71.43  percent  of 
DAFSC  2816,  70.59  percent  of  DAFSC  2724  and  64.2  percent  of 
DAFSC  28X5  selected  0  to  2  hours  as  the  amount  of  PDR  time 
that  should  be  devoted  to  system  overview  and  tutorial. 


DAFSC 

TABLE  59 

vs  PDR  Tutorial  Hours 

DAFSC 

0-2 

3-5 

6-9 

10  PLCJS 

2716 

58.33 

36.11 

2. ■’8 

2.78 

2816 

71.43 

21.43 

7.14 

2724 

70.59 

29.41 

— 

28X5 

64.20 

29.63 

4.94 

1.23 
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Investigative  question  14b.  Should  there  be  a  system 
overview  or  tutorial  as  part  of  a  CDR  and  how  much  of  a  COR 
should  be  devoted  to  system  overview  and  tutorials? 

Discussion.  Survey  question  24  asked  how  many 
hours  of  a  CDR  should  be  devoted  to  system  overview  and 
tutorial  information.  Survey  questions  6  and  9  were  used  to 
gather  the  responses  to  question  24  by  the  group  of 
individuals  having  participated  in  a  CDR  and  identifing 
responses  against  the  DAFSC. 

Findings.  Of  the  72.4  percent  of  the  respondents 
having  participated  in  CDRs;  66.86  (Table  4.60)  percent 
indicated  less  than  2  hours  of  a  CDR  should  be  devoted  to 
system  overview  and* tutor ial ,  25.29  percent  claimed  3  to  5 
hours,  5.75  percent  claimed  5  to  9  hours  and  2.3  percent 
claimed  10  or  more  hours. 

Fifty-seven  percent  of  DAFSC  2716,  75  percent  of  DAFSC 
2816,  78.79  percent  of  DAFSC  2724  and  62.5  percent  of  DAFSC 
28X5  selected  0  to  2  hours  as  the  amount  of  CDR  time  that 
should  be  devoted  to  system  overview  and  tutorial. 


TABLE  60 

DAFSC  vs  CDR  Tutorial  Hours 


DAFSC 

fM 

1 

CB 

3-5 

6-9 

10  FLU 

2716 

57.58 

36.36 

6.06 

— 

2816 

75.00 

17.86 

7.14 

2724 

78.79 

21.21 

— 

28X5 

68.50 

25.00 

7.50 

5 .30 
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Investigative  question  15a.  What  correlation  exist 
between  an  individual's  opinion  of  the  amount  of  system 
overview  required  to  the  amount  conducted  on  the  last  PDR? 

Discussion.  Survey  questions  16  and  30  were  used 
to  determine  the  correlation  between  the  amount  of  time  an 
individual  claimed  should  be  devoted  to  PDR  system  overview 
and  amount  actually  presented  at  the  respondents  last  PDR. 
SAS  Proc  Corr  was  used  to  compute  the  correlation  factor.  A 
strong  correlation  to  this  question  would  indicate  the 
amount  of  system  overview  presented  at  a  PDR  is  in  alignment 
with  the  perceived  amount  of  system  overview  time  required 
in  a  PDR. 

Findings.  Sixty-five  percent  of  the  individuals 
claimed  less  than  2  hours  should  be  devoted  to  system 
overview.  Sixty-eight  and  26.72  percent  of  these  indicated 
less  than  2  hours  and  3  to  5  hours  of  the  last  PDR  were 
devoted  to  system  overview,  respectively. 

Twenty-nine  percent  of  individual  claimed  3  to  5  hours 
should  be  devoted  to  system  overview.  Forty-two  and  38.46 
percent  of  these  indicated  less  than  2  hours  of  the  last  PDR 
were  devoted  to  system  overview,  respectively.  There  was  a 
significant  'correlation  of  .449  between  what  the  individual 
claimed  should  be  devoted  toward  system  overview  and  how 
much  was  devoted  to  system  overview  on  individual's  last 
PDR. 
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TABLE  61 

Recommended  vs  Actual  PDR  Overview 
Survey  Question  16  vs  Survey  Question  30 


Perceived 
Amount  of 
Overview 
(hours) 

Amount  of  Overview  Last  PDR 
(Hours) 

10 

0-2  3-5  6-9  or  More 

Total 

0-2 

63.97 

26.72 

3.45 

0.86 

65.17 

3-5 

42.31 

38.46 

15.38 

3.85 

29.21 

6-9 

12.50 

37.50 

37,50 

12.50 

4.49 

10  or 

More 

■“  — 

0.56 

0.56 

1.12 

Investigative 

question  15b. 

What 

correlation 

exists 

k 

between  an  individual's  opinion  of  the  amount  of  system 
overview  required  to  the  amount  conducted  on  the  last  CDR? 


Discussion.  Survey  questions  24  and  38  were  used 
to  determine  the  correlation  between  the  amount  of  time  an 
individual  claimed  should  be  devoted  to  CDR  system  overview 
and  amount  actually  presented  at  the  respondents  last  CDR. 
SAS  Proc  Corr  was  used  to  compute  the  correlation  factor.  A 
strong  correlation  to  this  question  would  indicate  the  amont 
of  system  overview  presented  at  a  CDR  is  in  alignment  with 
the  perceived  amount  of  system  overview  time  required  in  a 
CDR. 

Findings .  Sixty-six  percent  of  the  individuals 
claimed  less  than  2  hours  should  be  devoted  to  system 
overview.  Seventy-six  and  19.47  percent  of  these  indicated 
less  than  2  hours  and  3  to  5  hours  of  the  last  CDR  were 
devoted  to  system  overview,  respectively. 


Twenty-four  tercent  of  individual  claimed  3  to  5  hours 
should  be  devoted  to  system  overview.  Fifty  and  38.14 
percent  of  these  indicated  less  than  2  hours  of  the  last  CDR 
were  devoted  to  system  overview,  respectively.  There  was  a 
significant  correlation  of  .481  between  what  the  individual 
claimed  should  be  devoted  toward  system  overview  and  how 
much  was  devoted  to  system  overview  on  the  individual's  last 
CDR. 


TABLE  62 

Recommended  vs  Actual  CDR  Overview 
Survey  Question  24  vs  Survey  Question  38 


Perceived 
Amount  of 
Overview 
(hours) 

Amount  of  Overview  Last 
(Hours) 

0-2  3-5  6-9 

CDR 

10 

or  More 

Total 

0-2 

76.11 

19.47 

4.42 

— 

66.86 

3-5 

50.00 

38.10 

9.52 

2.38 

24.85 

6-9 

20.00 

30.00 

20.00 

30.00 

5.92 

10  or 

25.00 

50.00 

25.00 

2.37 

More 

Investigative  question  16.  Were  AF  participants 
required  to  state  their  individual  PDR  and  CDR  objectives? 

Discussion.  Survev  question  49  asked  if  during 
the  Air  Force  meeting  prior  to  the  PDR  and  CDR  if  all  AF 
participants  were  required  to  state  their  individual  PDR  and 
CDR  objectives. 


Findings. 

An  overwhelming  77 

.8  percent 

indicated 

the  AF  participants 

were  not  required  to 

state  their 

individual  PDR  and 

CDR  objectives  prior 

to  the  PDR 

or  CDR. 

Twenty-one  percent 

indicated  they  were  required  to 

state 

their  objectives. 

TABLE  63 

Survey  Question  49:  Participants 
Objectives  Stated 

Response 

#  of 

Respondents 

%  of 

Respondents 

Yes 

47 

21.8 

NO 

168 

77.8 

Did  Not  Know 

1 

0.5 

Total 

216 

100.0 

PDR/CDR  Training  and  Guidance 

Tables  64  through  69  covers  the  category  of  PDR  and  CDR 
training  and  guidance. 

Investigative  questions  seventeen  through  twenty  posed 
in  chapter  one  are  addressed  here. 

Invest ~  v-tive  question  17.  What  type  of  PDR  and  CDR 
training  and  guidance  has  the  individual  team  member  had? 

Discussion.  Survey  question  43  asked  if  the 
individual  had  any  guidance  on  how  to  conduct  himself  during 
a  PDR  or  CDR,  and  if  so,  was  it  self  taught,  training,  etc. 
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Cross  tabulation  of  the  survey  question  and  specific  DAFSC 
were  examined. 

findings.  Fifty-three  percent  of  all  respondents 
indicated  they  had  some  type  of  guidance  or  training  on  how 
to  conduct  themselves  during  PDRs  and  CDRs.  Forty-six 
percent  of  all  respondents  indicated  they  did  not  have  any 
guidance  training. 

Seventy  percent  of  DAFSC  2716,  52.63  percent  of  DAFSC 
2816,  57.45  percent  of  DAFSC  2724,  and  only  45.69  percent  of 
DAFSC  28X5  indicated  having  some  type  of  guidance  and 
training  on  how  to  conduct  themselves  at  PDRs  and  CDRs. 


TABLE  64 

DAFSC  vs  Prior  Review  Training 


DAFSC 

YES 

NO 

2716 

70.45 

29.55 

2816 

52.63 

47.37 

2724 

57.45 

42.55 

28X5 

45.69 

54.31 

Table  65  summarizes  some  of  the  training  and  guidance 
received  by  the  respondents.  The  table  shows  the  most 
frequently  mentioned  guidance  by  DAFSC. 

The  most  common  guidance  were  self  taught,  on  job 
training,  observation  and  Mil  Std  1521. 

Additional  comments  to  this  question  are  included  in 
the  Appendix  C:  Selected  Comments. 
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TABLE  65 

Training/Guidance  Racaivad 
OAFSC  TYPE 


2716  Saif  Taught 

AF  Regulations 
Oicact  supatvision 
DSMC 

Inhouse  Training 
PMD 

AFIT  short  courses 
Obsarvation 
Group  discussions 

2724  Self  taught 

On  Job  Training 

Instruction  from  engineering  personnel 
direct  supervision 
Mil  Std  1521 

2816  Mil  Std  1521 

Self  taught 
On  Job  Training 

Attending  other  related  system  PDR/CDRs 

Observation 

DSMC 

28X5  Self  taught 

By  doing  and  asking  others 

On  Job  Training 

Observing 

In  house  training 

Mil  Std  1521 

Short  courses 


Investigative  question  18.  Would  initial  PDR/CDR 
training  be  useful? 

Discussion.  Survey  question  53  asked  the 
individual  if  initial  PDR/CDR  training  would  be  useful. 
Survey  question  9  and  53  together  address  this  investigative 
question  by  DAFSC. 


Finding*.  Eighty-six  peccant  of  all  respondents 
indicated  initial  training  would  be  useful*  Only  6.7  percent 
indicated  initial  training  would  not  be  useful  and  5.4 
percent  did  not  know. 


TABLE  66 

Survey  Question  53:  Initial  Review  Training 


Response 

Percentage  of  Respondents 

Yes 

86.2 

No 

5.4 

Did  Not  Know 

6.7 

Not  Important 

0.8 

Total 

97.1 

Investigative  question  19a.  How  much  acquisition 
experience  should  an  individual  have  before  participating  in 
a  PDR? 

Discussion.  Survey  question  54  asked  the 
respondent  how  many  months  of  acquisition  experience  should 
one  have  before  participating  in  a  PDR.  Only  individuals 
having  participated  in  PDR  were  examined.  The  response  to 
the  survey  question  was  examined  against  the  individual's 
DAFSC. 

Findings.  Table  67  shows  the  distribution  by 
DAFSC  of  the  number  of  months  of  acquisition  experience  an 
individual  should  have  before  participating  in  a  PDR.  The 
five  most  frequent  indications  were  30.72  percent  stating  6 
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months,  28.31  percent  stating  12  months,  8.43  percent 
stating  8  months,  7.83  percent  stating  3  months  and  6.63 
percent  stating  24  months  of  acquisition  experience  needed 
before  participating  in  a  PDR. 

Forty-one  percent  of  DAFSC  2716,  claimed  12  months  of 
acquisition  experience  was  needed  before  participating  in  a 
PDR.  Twenty-two  percent  of  DAFSC  2716  claimed  6  months. 
Thirty-eight  percent  of  DAFSC  2724  claimed  6  months.  Sixteen 
percent  of  DAFSC  2724  claimed  3  months  and  another  16.13 
percent  claimed  12  months.  Twenty-one  percent  of  DAFSC  2816 
claimed  6  months.  Seventeen  percent  of  DAFSC  2816  claimed  0 
months.  Fourteen  percent  of  DAFSC  2816  claimed  3  months  and 
another  14.29  percent  claimed  12  months.  Thirty-three 
percent  of  DAFSC  28X5  claimed  6  months  and  another  33.33 
claimed  12  months.  Eight  percent  of  DAFSC  28X5  claimed  24 
months. 

TABLE  67 

DAFSC  vs  Acquisition  Experience  Recommended 
Before  PDR  Participation 

Number  of  Months  Recommended 


DAFSC 

0 

3 

6 

12 

24 

2716 

9.68 

3.23 

25.58 

41.94 

6.45 

2816 

17.86 

14.29 

21.43 

14.29 

10.71 

2.724 

3.23 

16.13 

38.71 

16.13 

— 

28X5 

6.67 

4.00 

33.33 

33.33 

8.00 

Total 

8.37 

7.83 

30.72 

28.31 

6.63 

34 


Investigative  question  19b,  How  much  acquisition 
experience  should  an  individual  have  before  participating  in 
a  COR? 

Discusf ion.  Survey  question  54  asked  the 
respondent  how  many  months  of  acquisition  experience  should 
one  have  before  participating  in  a  CDR,  Only  individuals 
having  participated  in  CDR  were  examinded.  The  response  to 
the  survey  question  was  examined  against  the  individual's 
DAFSC. 

Findings.  Table  68  shows  the  distribution  by 
DAFSC  of  the  number  of  months  of  acquisition  experience  an 
individual  should  have  before  participating  in  a  CDR.  The 
five  most  frequent  indications  were  26.22  percent  stating  12 
months,  25  percent  stating  6  months,  12.2  percent  stating  24 
months,  10.37  percent  stating  18  months  and  8.54  percent 
stating  <ii  months  of  acquisition  experience  needed  before 
participating  in  a  CDR. 

Forty-one  percent  of  DAFSC  2716  claimed  12  months  of 
acquisition  experience  was  needed  before  participating  in  a 
CDR.  Twenty  percent  of  DAFSC  2716  claimed  18  months. 

Thirteen  percent  of  DAFSC  2716  claimed  6  months.  Thirth-one 
percent  of  DAFSC  2724  claimed  6  months.  Eighteen  percent  of 
DAFSC  2724  claimed  12  months.  Twelve  percent  of  DAFSC  2724 
claimed  3  months.  Twenty-five  percent  of  DAFSC  2816  claimed 
6  months.  Seventeen  percent  of  DAFSC  2816  claimed  0,  12  and 
24  months,  respectively.  Twenty-six  percent  of  DAFSC  28X5 


claimed  6  and  12  months, respectively.  Fourteen  percent  of 


DAFSC  28X5  claimed  24  months. 


TABLE  68 

DAFSC  vs  Acquisition  Experience  Recommended 
Before  CDR  Participation 

Number  of  Months  Recommended 


DAFSC 

• 

6 

12 

18 

24 

2716 

18.34 

13.79 

41.38 

28.69 

6.98 

2816 

17.86 

25.18 

17.86 

3.57 

17.86 

2724 

3.13 

31.25 

18.75 

9.37 

6.25 

28X_ 

6.67 

26.67 

26.67 

9.33 

14.67 

Total 

8.54 

25.88 

26.22 

18.37 

12.28 

Investigative  question  28a.  What  is  the  single  most 
useful  guide  for  PDR  procedural  guidance? 

Discussion.  Survey  question  55  asked  the 
respondents  to  comment  on  what  the  single  most  important 
guide  (ex.  regulation,  manual  or  standard)  is  for  PD’% 
prepart ion. 

The  responses  of  those  who  had  participated  ’n  PDR  were 
examined . 

Findings.  The  response  of  those  individuals 
having  participated  in  a  PDR  are  summarized  and  grouped  by 
DAFSC  in  the  Table  69. 

The  three  most  common  responses  across  all  DAFSCs  was 
the  Mil  Standard  1521,  job  experience  and  DSMC  handbook 
Systems  Engineering  Management  Guide. 
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Additional  comments  to  this  question  ate  included  in 


Appendix  c:  Selected  Comments. 


Investigative  question  28b.  What  is  the  single  most 
useful  guide  foe  CDR  ptocedutal  guidance? 

Discussion.  Survey  question  55  asked  the 
respondents  to  comment  on  what  the  single  most  important 
guide  (ex.  regulation,  manual  or  standard)  is  for  CDR 
prepart ion. 

The  responses  of  those  who  had  participated  in  CDR  were 
examined. 

Findings.  The  response  of  those  individuals 
having  participated  in  a  CDR  are  summarized  and  grouped  by 
DAFSC  in  Table  €9. 

The  three  most  common  responses  across  all  DAFSCu  was 
the  Nil  Std  1521,  job  experience  and  the  DSMC  handbook. 
System  Bngineering  Management  Guide. 

Additional  comments  to  this  question  are  included  in 
Appendix  C:  Selected  Comments. 
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DAFSC 

2716 


2724 


2816 

28X5 


TABLB  69 

PDR/CDR  Guidance  Recommended 
TYPE 


AF  Regulations 
Nil  Std  1521 

DSNC  System  Engineering  Management  Guide 

Group  Meetings  within  SPO 

PMD 

AF  Regulation  810-3 
Mil  Std  1521 

DSMC  System  Engineering  Management  Guide 
Mil  Std  1521 

DSMC  System  Engineering  Management  Guide 

Project  Officers  Handbook 
Mil  Std  1521 

DSMC  System  Engineering  Management  Guide 


V.  Conclusions  and  Recommendations 


Overview 

Success  of  s  review  depends  on  both  Government  snd 
contractor  preparation  before  the  meeting  (3:13-9). 

Webster  (16:9)  states,  "The  staff  meeting  is  a  time  to 
report  on  your  homework,  not  to  do  it." 

The  effectiveness  of  technical  meetings  or  design 
reviews  can  be  seen  through  the  results  of  design  reviews 
evaluated  based  on  the  number  of  established  action  items, 
frequency  of  design  changes  and  post  review  investigations 
(10:213) • 

Review 

This  chapter  presents  the  conclusions  and1 
recommendations  that  can  be  drawn  from  this  research  effort. 
Although,  the  survey  response  rate  was  low  it  could  easily 
support  a  confidence  level  between  85  and  90  percent.  This 
confidence  level  could,  for  opinions  and  attitudes,  allow 
for  inference  of  the  research  results  to  the  entire  Air 
Force  population. 

The  major  limitation  of  the  research  was  the  exclusion 
of  civil  service  employees  and  contracting  personnel. 

The  twenty  investigative  questions  examined  three  major 
areas  of  PDRs  and  CDRs:  the  purposes  as  stated  in  Nil  Std 
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1521,  PDR/CDR  effectivcness/eff iciency,  and  PDR/CDR  training 
and  guidance. 

The  following  tactions  diacutt  conclusions  drawn  from 
the  results  presented  in  chapter  four. 

Military  Standard  1521  PDR/CDR  Purposes.  The  results 
of  investigative  questions  one  through  three  pertain  to  this 
area  of  research. 

The  results  of  the  research  indicated  that  IDR  and  COR 
tend  to  be  conducted  in  accordance  with  Mil  Std  l.t21,  with 
the  exception  of  providing  adequate  coverage  of  the 
technical  risk  associated  with  the  manufacturing  Methods  and 
processes,  and  adequate  coverage  of  the  physical  and 
functional  interfaces  among  the  configuration  itesis,  other 
equipment,  facilities  and  computer  programs. 

Those  respondents  who  have  and  have  not  participated  in 
PORs  and  CDRs  tend  to  agree  with  the  stated  PDR  and  CDR 
purposes  of  Mil  Std  1521.  The  experience  factor  did  noc 
significantly  impact  the  individual's  opinion  of  the  stated 
PDR/CDR  purposes. 

PDR/CDR  Ef feet iveness/Bf f iciency .  The  results  of 
investigative  questions  four  through  sixteen  pertain  to  this 

area. 

Over  €•  percent  of  the  respondents  indicated  the  PDRs 
and  CDRs  they  participated  in  last  were  approved  contingent 
upon  some  action  to  be  completed.  Only  0.6  percent  of  the 
repondents  indicated  the  last  PDR  they  participated  in  was 
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disapproved#  and  only  2.8  percent  said  the  sane  of  their 
last  CDR. 

Before  erasing  theue  questions  it  is  interesting  to 
note  that  over  78  percent  of  the  respondents  felt  that  the 
last  PDR  provided  adequate  evaluation  of  the  system  to  take 
the  next  step  in  the  acquisition  process. 

Over  (B  percent  felt  the  last  CDR  provided  adequate 
evaluation  of  the  system  to  proceed  into  the  next 
acquisition  phase. 

However#  less  than  48  percent  of  the  respondents 
indicated  that  all  POR  and  CDR  action  items  were  resolved 
before  the  PDR  and  CDR  approval  was  given. 

Over  65  percent  indicated  that  at  the  conclusion  of  the 
PDR  and  CDR  sooM  action  items  remained  open. 

Of  the  AZs  presented  at  the  PDR  and  CDR#  some  already 
existed  prior  to  the  PDR/CDR  as  indicated  by  79.13  percent 
and  86.78  percent  for  the  survey  respondents,  respectively. 

Program  managers  tend  to  believe  that  all  action  items 
should  be  resolved  before  entering  a  PDR#  whereas# 
development  engineers  tend  to  be  less  concerned  about  action 
items  being  completed.  Both#  the  majority  of  program 
managers  and  development  engineers  indicated  all  action 
items  should  be  resolved  before  entering  a  CDR. 

Although#  the  respondents  felt  all  action  items  should 
be  resolved  before  entering  a  CDR#  less  than  40  percent  of 
the  respondents  indicated  this  to  have  taken  place  on  their 
last  program  CDR  attended. 
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Over  Si  percent  of  ell  respondents  indicated  there  were 
slippages  in  both  system  design  and  development  and  CDR 
schedules. 

Over  64  percent  felt  that  the  system  design  should  be 
complete  prior  to  approving  a  CDR. 

Forty-six  percent  claimed  major  design  problems  and 
modifications  were  presented  at  the  last  CDR  they 
participated  in  and  of  these,  69. S  percent  indicated  the 
design  problems  were  covered  in  previous  meetings. 

Examination  of  the  respondents  overall  opinion  of  items 
to  be  complete  before  entering  a  CDR  includes  resolutions  of 
all  action  items  and  a  complete  design.  In  addition,  it  was 
felt  by  the  majority  of  respondents  that  all  action  items 
issured  during  a  PDR  or  CDR  should  be  resolved  before  the 
meeting  is  given  an  approval  rating.  Also,  no  major  design 
modifications  and  problems  should  be  presented  at  a  CDR, 
especially  if  they  had  existed  previously. 

However,  the  trend  of  the  respondents  showed  their  last 
PDR  and  CDR  to  be  just  the  opposite  of  what  they  felt  should 
had  been  the  case  (i.e.  action  items  resolved,  no  major 
design  problems  and  modifications  presented). 

Forty-nine  percent  claimed  there  was  inadequate  time  to 
review  the  CDR  data  package  and  44.8  percent  of  the 
respondents  claimed  the  data  was  not  delivered  on  schedule. 
This  leads  to  inadequate  and  insufficient  review  of  the  data 
and  PDR/CDR  objective  preparation. 
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Majority  of  respondents  fait  there  should  ba  some 
system  overview  or  tutorial  prasantad  at  PDRs  and  CDRs,  but 
most  indicatad  no  more  than  two  hours,  However,  tha  trand 
for  tha  last  PDR  a  id  CDR  rangad  from  lass  than  two  hours  to 
nina  hours  for  a  PDR  and  lass  than  two  hours  to  fiva  hours 
for  a  CDR. 

Seventy-seven  par cant  of  tha  raspondants  indicatad  PDR 

* 

and  CDR  participants  wars  not  required  to  stata  thair 
individual  objactivaa.  This  is  tha  largast  indication  of  how 
ill  praparad  hW  participants  ara  for  upcoaing  PDRa  and  CDRs. 

Another  alaraing  concorn  is  tha  fraquancy  of  "Did  not 
know"  rasponsas  for  survay  quastions  supporting 
invast igativa  quastions  four  through  sixteen.  For  axaapla 
lt.S  and  8.5  parcant  did  not  know  what  approval  rating  was 
givan  at  a  PDR  and  CDR,  raspactivaly.  Savanty  and  65.79 
parcant  of  thosa  not  having  participatad  in  PDRs  and  CDRs 
did  not  know  if  tha  apacific  aaating  adaquataly  avaluatad 
tha  systea  bafora  procaading  with  tha  program.  Siailar 
parcantagas  axistad  for  tha  survay  quastions  addrassing 
action  itaas,  dasign  problems  and  modifications,  system 
dasign/davalopaent  and  CDR  schedule  slippages,  CDR  data 
package  and  systaa  ovarviaw  issues. 

The  conclusion  is  that  individuals  could  only  respond 
to  tha  survay  quastions  addrassing  thaa  directly  versus 
program  issues. 

Coaplete  understanding  of  the  prograa,  its  direction 
and  overall  importance  of  PDR  and  CDR  seemed  to  be  lacking. 
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PPR/CPR  Training  and  Guidance.  The  results  of 
ivestigative  questions  seventeen  through  twenty  pertain  to 
this  area  of  research, 

Ixcept  for  the  development  engineers#  company  grade 
officers  (over  31  percent  in  each  DAFSCs)  claiaed  to  have 
had  some  foraer  training  and  guidance  on  PDft/CDR  procedures. 
Field  grade  prograa  aanagers  overwhelmingly  (7t.4S  percent) 
claiaed  to  have  had  previous  PDR/CDR  training  and  guidance. 
The  aost  coaaon  types  were  self  taught#  on  job  training# 
observation  and  Mil  Std  1321. 

Over  86.2  percent  of  all  respondents  felt  initial 
training  would  be  useful. 

On  the  issue  of  acquisition  experience  aost  respondents 
felt  6  to  12  aonths  should  be  required  before  participation 
in  a  PDR  and  CDR. 

The  useful  guidances  recoaaended  by  the  respondents 
aost  often  wes  Mil  Std  1521#  prior  experience  and  DSMC's 


Recommendations 

As  stated  previously#  in  order  to  make  inferences  to 
the  entire  population  working  PDRs  and  CDRs#  the  civil 
service  and  contractor  personnel  should  be  surveyed. 

to  those  individuals  preparing  for  a  PDR/CDR  consider 
the  following: 

1.  Barly  preparation  and  identifications  of  individual 
objectives  is  of  the  upmost  importance. 
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2*  individuals  should  read  and  uss  Mil  Std  1521  and 
OSMC'a  System  Bnqi nesting  Management  Guide. 

3*  Bach  parson  should  hava  at  laast  6  months 
acquisition  axparianca  prior  to  participating  in  a  PDR  and 
CDR. 

4.  Bach  parson  should  undarstand  tha  significanca  of 
action  i tarns,  system  overviews,  astablishing  solid 
objactivas  and  PDR/CDR  praparaticn. 

5.  Tha  program  should  not  start  CDR  with  major  design 
problams  in  tha  works  or  significant  action  itams 
unrasolvad. 

6.  CDRs  should  not  ba  approved  with  unrasolvad  action 
itams  and  design  issues. 

7.  Barly  in  tha  program  stages  a  CDR  data  delivery 
schedule  should  ba  established  according  to  program  sise, 
review  team  size  and  data  package  sise. 

8.  If  data  has  not  been  completely  reviewed  and 
responses  satisfactorily  addressed,  strong  consideration 
should  be  given  to  slipping  the  CDR  start  date. 

9.  PDRs  and  CDRs  should  be  more  than  contractually 
binding  to  be  held,  but  to  be  satisfactorily  completed  with 
all  significant  action  items  and  design  problems  resolved 
prior  to  a  satisfactory  approval. 

IB.  Follow  the  stated  purposes  of  Mil  Std  1521. 

11.  Minimize  the  percentage  of  a  CDR  devoted  to  systems 
overview  and  tutorial. 
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12.  Have  each  AF  participant  identify  his  objectives 
met  at  the  conclusion  o*  each  day's  meeting  and  also  define 
his  objectives  fc  the  remainder  of  the  meeting. 

Summary 

PDR's  and  CDR's  are  two  of  the  most  critical  technical 
meetings  of  a  program  which  consumes  large  expenditures  of 
time,  money  and  resources  from  the  AF  and  contractors. 

The  results  of  this  research  indicate  most  PDRs  and 
CDRs  are  not  as  effective  as  they  could  be.  Two  of  the 
primary  reasons  are  the  lack  of  knowledge  on  what  should  be 
accomplished  by  participants  and  the  lack  of  sufficient 
contractual  importance  for  complete  and  succet  .1  PDRs  and 
CDRs. 

Follow  on  research  suggestions 

Three  specific  areas  for  follow  on  research  are: 

1.  Determine  the  opinions  and  attitudes  of  the  civil 
service  and  contractor  personnel  on  PDR  and  CDR  procedures 
and  effectiveness. 

2.  Evaluate  the  cost  for  the  AF  and  the  contractor  to 
put  on  a  PDR  and  CDR. 

3.  Determine  specific  inhouse  PDR/CDR  training  that 
could  be  used  by  program  managers  and  development  engineers. 
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Appendix  At  Questionnaire  and  Instructions 

DCPAATMCNT OP THS  AM POftCS  USAF  SCN  87*58 


mi  ranee  Mmun  on  ncuwoioaT 

mtOMT-n*TTtMOM  Ml  *0*01  SMS0M4MI 


LAY  (Copt  Bennett,  AOTOVOM  715-4569) 

Preliminary  and  Critical  Ooaign  RotIm  Procedures  and 
If fectiveness  Survey  Package 


1.  Pleaae  take  the  time  to  coaplete  the  attached  questionnaire 
and  return  it  in  the  oncloaod  envelope  by  29  Nay  1997. 

2.  The  aurvey  aeaaurea  your  perceptiona  and  attitudea  toward 
Preliminary  and  Critical  Oeaign  Review  procedures  and  their 
effectiveness  in  evaluating  systeai  design  and  developatent  pro¬ 
gress.  The  data  we  gather  will  becoaM  part  of  an  APIT  research 
project  and  any  influence  Preliminary  and  Critical  Design  Review 
procedure  changes.  Your  individual  responses  will  be  combined 
with  others  and  will  not  be  attributed  to  you  personally. 

3.  Your  participation  is  completely  voluntary,  but  we  would 
certainly  appreciate  your  help,  for  further  information  please 
contact  Capt  Bennett  at  AOTOVOU  755-6569. 


Uh  DOMOMD,  Lt  Col,  OSAF 
^Tead,  Department  of  System 
Acquisition  Management 
School  of  Systems  and  Logistics 


2  Ateh 

1 .  Survey 

2.  Return  Bnvelope 


j  mt^  a  m-u  w-vi  wu  ru  mvi  v~v  ru  anJ  jrv  W'JifV.jrw  RV  M-d  NhJ  M.xi  .N-'*  N‘-  M  M'kMtV  W 


SURVEY  PRELIMINARY  AND  CRITICAL  DESIGN  REVIEWS 


Part  I  -  Background  Information 

1.  Military  Rank  : _ 

2.  Office  Symbol: _ 

3.  With  which  acquisition  phase  are  you  most  familiar? 

(check  the  appropriate  ones) 

o  Concept  Exploration  o  Full  Scale  Development 

o  Demonstration/Validation  o  Production 

4.  With  which  type  of  acquisition  are  you  most  familiar? 

(check  the  appropriate  ones) 
o  Aircraft  o  Space/Missile 

o  Armament  o  Other  (specify) 

o  Electronics  _ _ 


5.  Number  of  ye^rs  of  acquisition  experience  you  have  ? 

(check  one) 

o  0-2  years  o  8-10  years 

o  3-4  years  o  11-14  years 

o  5-7  years  o  more  than  15  years 

6.  Number  of  Preliminary  and  Critical  Design  Reviews 
(System 

level)  in  which  you  have  formally  participated  in: 
PDR _ (number)  CDR _ (number) 

7.  Current  Functional  Area:  (check  one) 

o  Program/Project  Management 

o  Contracting/Manufacturing  Management 

o  Engineering 

o  Configuration  Management 

o  Logistics  Management 

o  Test  and  Evaluation 

o  Other  (please  specify) _ 

8.  Education  type  (i.e.  BS  Electrical  Engineering): 


9.  Duty  AFSC: 
o  2716 

o  2816 


(check  one) 
o  .2724 
o  28X5 
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Pact  II  -  Purpose  o£  the  Preliminary  Design  Review  (PDR) 

Questions  10-14  require  a  response,  ranging  from  strongly 
agree  to  strongly  disagree,  concerning  your  opinion  on  the 
purpose  of  a  PDR. 


RESPOND 

l  TO  QUESTIONS  AS 

FOLLOWS : 

Strongly 

agree 

Agree 

Neither  agree 
nor  disagree 

Disagree 

Strongly 

disagree 

1 

2 

3 

4 

5 

10.  To 

evaluate  the 

progress,  technical  adequacy. 

and  risk 

resolution  (on  a  technical,  cost  and  schedule  basis)  of  the 
selected  design  approach. _ 

11.  To  determine  the  design  approach  compatibility  with 

performance  and  engineering  specialty  requirements  of  the 
Configuration  Item  (Cl)  development  specification.  _ 

12.  To  assess  the  technical  risk  associated  with  the 

selected  manufacturing  methods/processes. _ 

13.  To  define  the  physical  and  functional  interfaces  among 

the  Cl  and  other  items  of  equipment,  facilities,  computer 
programs,  and  personnel. _ 

14.  To  provide  a  system  overview  and  tutorial  for  the  PDR 

part  icipants._ _ 

15.  Should  all  action  items  be  resolved  before  approving 
the 

PDR  ?  (check  one) 

o  yes  o  don't  know 

o  no  o  not  important 

16.  How  many  hours  of  the  PDR  should  be  devoted  to  system 
overview  and  tutorial  information?  (check  one) 

o  0-2  hours  o  6-9  hours 

o  3-5  hours  o  10  plus  hours 


Pact  III  -  Purpose  of  the  Critical  Design  Review  (CDR) 

Questions  17-22  caquica  a  response,  ranging  from  strongly 
agraa  to  strongly  disagree,  concerning  your  opinion  on  the 
purposes  of  a  CDR. 


RESPOND 

TO  QUESTIONS  AS 

FOLLOWS: 

Strongly 

Strongly 

Agree 

Neither  agree 

Disagree 

agree 

disagree 

. 

nor  disagree 

1 

2 

3 

4 

5 

IV.  To  determine  that  the  detailed  design  of  the 
configuration  item  under  review  satisfies  the  performance 
and  engineering  specialty  requirements  of  the  Cl  development 
specifications^ _ 

18.  To  establish  the  detail  design  compatibility  among  the 

Cl  and  other  items  of  equipment,  facilities,  computer 
programs . _ 

19.  To  assess  the  configuration  item  risk  areas  (on  a 

technical,  cost  and  schedule  basis). _ 

28.  To  assess  the  results  of  the  producibility  analyses 
conducted  on  system  hardware  design. _ 

21.  To  review  the  preliminary  product  specification. _ 

22.  To  review  major  design  modifications. _ 

23.  Should  all  action  items  be  resolved  before  approving 

the  CDR.  (check  one) 

o  yes  o  don't  know 

o  ‘  no  o  not  important 

24.  How  many  hours  of  the  CDR  should  be  devoted  to  system 
overview  and  tutorial  information,  (check  one) 

o  8-2  hours  o  6-9  hours 

o  3-5  hours  o  10  plus  hours 
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Part  IV  -  Last  or  currant  program  PDR  quastions 

IF  YOU  HAVE  NOT  PARTICIPATED  IN  A  PDR  SKIP  PART  IV. 

Quastions  25-29  require  a  rasponsa,  ranging  from  strongly 
agraa  to  strongly  disagree,  concarning  your  opinion  on  how 
tha  purposas  of  tha  last  or  currant  PDR  you  attended  wara 
satisf iad. 

RESPOND  TO  QUESTIONS  AS  FOLLOWS : 

Strongly  Agraa  Naithar  agraa  Disagraa  Strongly 
agraa  nor  disagraa  disagraa 

1  2  3  4  5 

25.  Tha  raviaw  adaquataly  coverad  tha  technical  adequacy  of 

tha  selected  design  approach. _ 

26.  The  raviaw  adaquataly  covered  tha  risk  resolution  (on  a 

technical,  cost,  and  schedule  basis)  of  the  selected  design 
approach. _ 

27.  Tha  raviaw  adaquataly  covered  the  design  approach 

capability. in  meeting  the  performance  and  engineering 
specialty  requirements  of  the  Cl  development 
specification. _ _ 

28.  The  review  adequately  covered  the  technical  risk 

associated  with  selected  manufacturing  methods  and 
processes . _ 

29.  The  review  adequately  covered  the  physical  and 

functional  interfaces  among  tha  Cl  and  other  equipment, 
facilities,  computer  programs,  and  personnel. _ 

30.  How  many  hours  of  the  PDR  wf»re  devoted  to  system 
overview  and  tutorial  information?  (check  one) 

o  0-2  hours  o  6-9  hours 

o  3-5  hours  o  10  plus  hours 

31.  Were  all  action  items  resolved  before  an  approval  of 
the  PDR  was  given?  (check  one) 

o  yes  o  don't  know 

o  no  o  not  important 

32.  What  was  the  approval  given  ?  (check  one) 

o  approved 

o  approved  contingent  upon  some  action  completed 

o  approved  with  a  wavier 

o  disapproved 

o  don't  know 


Part  V  -  Last  or  currant  program  CDR  quaationa 

IF  YOU  HAVE  MOT  PARTXCXPATBD  XN  A  CDR  SKIP  PART  V. 

Quaationa  33-37  raquira  a  raaponaa,  ranging  from  atrongly 
agraa  to  atrongly  disagree,  concarning  your  opinion  on  ho* 
tha  purpoaaa  of  tha  laat  or  currant  CDR  you  attended  were 
aatiaf iad. 

RESPOND  TO  QOSSTIONS  AS  FOLLOttS: 

Strongly  Agroa  Naithar  agree  Diaagraa  Strongly 
agraa  nor  diaagraa  diaagraa 

1  2  3  4  5 

33.  Tha  review  adequately  determined  that  tha  configuration 

item  under  review  satisfied  tha  performance  and  engineering 
apacialty  requirements  of  tha  Cl  development 
specification. _ 

34.  Tha  review  adequately  determined  that  the  detailed 

design  was  compatible  between  the  Cl  and  the  other  items  of 
equipment,  facilities,  and  computer  programs. _ 

35.  The  review  adequately  assessed  tha  Cl  risk  areas  (on  a 
technical,  cost  and  schedule  basis) . 

36.  The  review  adequately  assessed  the  producibility  of  the 

system  hardware  design. _ 

37.  The  review  adequately  covered  the  preliminary  product 

specification. _ 

38.  How  many  hours  of  the  CDR  were  devoted  to  system 
overview  and  tutorial  information,  (check  one) 

o  0-2  hours  o  6-9  hours 

o  3-5  hours  o  10  plus  hours 

39.  Were  all  action  items  resolved  before  an  approval  of 
the  CDR  was  given?  (check  3ne) 

o  yes  o  don't  know 

o  no  o  not  important 

40.  Were  there  any  major  design  modifications  presented?  If 

so,  approximately  how  many?  (check  one) 

o  yes  quantity _  o  don't  know 

o  no  o  not  important 

41.  What  was  the  CDR  approval  given?  (check  one) 

o  approved 

o  approved  contingent  upon  some  action  completed 

o  approved  with  a  wavier 

o  disapproved 

o  don't  know 
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Pact  VX  -  General  POP  and  COR  question* 

42.  The  COR  should  not  be  conducted  until  the  detailed 
design  of  each  configuration  item  la  complete,  (check  one) 

o  yes  o  no  o  don't  know  o  not  important 

43.  Have  you  had  any  guidance  on  how  to  conduct  yourself 
during  a  review?  If  so*  what  type  (self  taught,  training, 
education  or  direct  supervision)?  (check  one) 

o  yes  o  no 

describe i _ 

44.  On  the  last  or  current  prograe  you  attended  did  the  CDR 

schedule  slip  ?  If  so,  give  the  approximate  number  of 
weeks. _  (weeks)  (check  one) 

o  yes  o  no  o  don't  know  o  noc  important 

45.  On  the  last  o*  current  program  you  attended  were  there 
any  AIs  left  opened  at  the  conclusion  of  the  PDR  and  CDR  ? 

If  so,  how  many  ?  (check  one  for  PDR  and  CDR) 

PDR  CDR  PDR  CDR 

o  o  yes  o  o  don't  know 

o  o  no  o  o  not  important 

If  yes,  quantity  (PDR) _  (CDR) _ 

46.  Of  the  action  items  presented  at  PDR  or  CDR  did  these 
already  exist  as  previous  meeting  issues  ?  (check  one  for 
PDR  and  CDR) 

PDR  CDR  PDR  CDR 

o  o  yes  o  o  don't  know 

o  o  no  o  o  not  important 

47.  If  you  answered  yes  to  question  46,  answer  the 
following  question,  of  these  action  items,  were  they 
resolved  or  closed  prior  to  the  completion  of  the  PDR  or 
CDR  ?  (check  one  for  PDR  and  CDR) 

PDR  CDR  PDR  CDR 

o  o  yes  o  o  don't  know 

o  o  no  oo  not  important 

48.  Were  design  problems  presented  at  the  CDR  covered  in 
previous  meetings  ?  (check  one) 

o  yes  o  don't  know 

o  no  o  not  important 

49.  During  the  last  PDR  or  CDR  AF  only  meeting,  were  all  AF 
participants  required  to  verbalize  their  individual 
objectives  for  the  PDR  and  CDR  ?  (check  one  for  PDR  and  CDR) 

PDR  CDR  PDR  CDR 

o  o  yes  o  o  don't  know 

o  o  no  oo  not  important 


L 
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Sfl.  Did  you  have  adequate  tiae  to  review  tho  CDR  supporting 
data  packages  to  your  satisfaction  and  vara  your  coaaanta 
addrassad  sufficiantly  by  tha  contractor  prior  to  tha  CDR  ? 
If  not  explain.  (chock  one) 

o  yas  o  don't  know 

o  no  o  not  iaportant 

o  coaaants 

51.  Kara  tha  CDR  supporting  data  packagas  coaplata  and 
dalivarad  on  schadule  ?  (chock  ona  for  complete  and 
dalivarad) 

coaplata  dalivarad  coaplata  dalivarad 

o  o  yas  o  o  don't  know 

o  o  no  oo  not  iaportant 

52.  On  tha  last  or  currant  prograa  you  attandad  wara  thara 

slippagas  in  tha  systaa  and  subsystaa  dasign  and 
devalopaant  ?  If  so,  giva  tha  approxiaata  nuabar  of 
subsystaas  and  waaks  of  slippaga  in  tha  critical  path  of  tha 
schadula.  (chock  ona) 

o  yas  o  don't  know 

c  no  o  not  iaportant 

If  yas  #s”bsysteas _  tweaks _ 

53.  Would  initial  PDR  and  CDR  training  ba  usaful  ?  (chock 
ona) 

o  yas  o  don't  know 

o  no  o  not  iaportant 

54.  How  aany  aonths  of  acquisition  axparianca  should  ona 
hava  bafora  participation  in  a  PDR  and  a  CDR  ? 

PDR _  CDR _  (aonths) 

55.  in  your  opinion,  what  is  tha  singla  most  iaportant 
guida  (ax.  regulation,  annual,  standard)  for  praparing  for  a 
PDR  and  a  CDR.  (plaasa  ba  specific) 

PDR 

CDR'  “  '  ~ 


56.  In  your  opinion,  did  the  last  or  currant  program  PDR 
and  CDR  you  attended  adequately  evaluate  the  system  to  allow 
it  to  proceed  into  the  next  acquisition  phase? (check  one  for 
PDR  and  CDR) 

PDR  CDR  PDR  CDR 

o  o  yes  o  o  don't  know 

o  o  no  oo  not  iaportant 


57.  On  the  last  or  current  program  PDR  and  CDR  you  attended 
approximately  how  many  Air  Force  personnel  participated  in 
the  meeting  ?  (check  one  for  PDR  and  CDR) 

PDR  CDR  PDR  CDR 

o  o  less  than  10  o  o 

o  o  10-24  oo 


less  than  10 
10  -  24 


25  -  34 
35  or  more 


Appendix  it  SAS  Proof* 


OPTIONS  LXNESZSR-84) 

PROC  FORMAT) 

VALOR  CORRRMT 

."'DID  MOT  ANSWER* 
1"'FIRST  LIROTINAMT' 

2" 'CAPTAIN* 

3- 'MAJOR' 

4" 'LIEUTENANT  COLONEL' 
5- 'COLONEL '| 

VALOB  DOTY 

."'DID  NOT  ANSWER' 
1"'2716 ' 

2«' 2816' 

3«»2724 ' 

4"' 28X5' 

VALOR  COMMAND 

."'DID  NOT  ANSNER* 

1- 'AFSC' 

2"' AFLC' 

3"'TAC* 

4*' MAC' 

5" 'SAC* 

6" ' AFCMD 1 
7-'ATC‘ 

8«'AFCC 
9"' AO' 

18" 'HQ  OSAF/SAF' 
11»'AFSPACMD' 

12" ‘ OTHER' 

13"' AFO  .‘EC1 
14-' FSC*; 

VALUE  DEGREE 

.-'DID  NOT  ANSWER' 
J.-'BS  ENGR ' 

2-  ' BS  CHEMISTRY' 

3® ' BA  MATH' 

4  - ' BS  OTHER' 

5-  BA  BUSINESS' 

6«*BA  MANAGEMENT' 

7- ' BS  PSYCHOLOGY' 
R-'BA  OTHER' 

'  *'MS  ENGR' 

18" 'MS  CHEMISTRY' 

11-  'MS  MATH' 

12-  'MS  OTHER' 

13-  ' MBA' 

14-  'MS  PSYCHOLOGY' 

15-  '  MA  OTHE  .* 

16-  ' FHD' ) 
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VALOR  YEARS 

.•'DID  HOT  ANSWER* 

1-*-  TO  2  !M* 

2*'3  TO  4  YES' 

3- 'S  TO  7  YR81 

4- '8  TO  19  YXI* 

3**11  TO  14  YRS’ 

4 ■'HORS  THAN  IS  YES' } 

VALOR  L1KIR 

.-'DID  HOT  ANSWER' 

1-' STRONGLY  AQRKI ' 

2* 1 MODERATELY  AGRII ' 

3-' NEITHER  AQRI1/DX8AQRRI 1 

5-  *  8TROHQLY  DX8AQRII* 

VALOR  PNASB 

.-•DID  HOT  AN8WER* 

1- *C/B' 

2-  ' D/V' 

3- * PSD' 

4** PROD* | 

VALUE  TYPE 

.-•DID  HOT  ANSWER* 

1- * AIRCRAFT' 

2- * ARMAHRHT' 

3* ' ELECTRON ICS * 

4-  ' SPACE/MILLILK ' 

5- * OTHER* | 

VALOR  FOHCT 

.-'DID  HOT  AH8NBR ' 

1-  ' PROGRAM/PROJECT  MANAGEMENT * 

2 -  ' CONTRACT I NG/MANOFACTOR I NG  MANAGEMENT  * 

3-  'ENGINEERING* 

4-  'CONFIGORATION  MANAGEMENT* 

5-  *  LOGaSTICS  MANAGEMENT* 

6-  'TEST  AND  EVLAOATION* 

7-  'OTHER*  ; 

VALUE  YESNO 

.-'DID  NOT  ANSWER* 

1-  ' YES  * 

2-  *  NO* 

VALUE  FENSE 

.-'DID  NOT  ANSWER* 

1- 'YBS* 

2-  ' NO' 

3-  'DO  NOT  KNOW 

4-  'NOT  IMPORTANT' ; 

VALUE  OVERVU 

.-'DID  NOT  ANSWER* 

1- 'i  TO  2  HOURS' 

2-  '3  TO  5  HOURS' 

3-  '5  TO  9  HOURS' 

4-  *10  OR  MORE  HOURS'; 

VALUE  QTYPERS 

.-'DID  NOT  ANSWER' 
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l-'LEfS  THAU  19  PARTICIPANTS' 

2*' If  TO  24  PARTICIPANTS • 

3-  '25  TO  34  PARTICIPANTS' 

4-  *35  OR  NORI  PARTICIAPNTS* ) 

VALOR  OJT 

.•'OIO  ROT  ANSWER' 

I- 'l  MONTHS' 

2* *2  MONTHS' 

3-'3  MONTH!' 

4*' 4  NONTNA' 

5- '5  MONTHS' 

€•'4  NORTHS' 

7«'7  MONTHS ' 

MONTHS' 

9-' 9  MONTHS' 
li« *19  MONTHS' 

II-  'll  MONTHS' 

I2-*12  MONTHS' 

I3«'N0RR  THAN  12  MONTHS') 

VALOR  APPROVAL 

••'DID  NOT  AN8NIR ' 

1- ' APPROVED ' 

2-  'APPROVED  CONTI NQINT' 

3-  'APPROVED  WITH  WAIVIRS* 

4-  'DISAPPROVED* 

5-  'DO  NOT  KNOW') 

DATA  INIT) 

INPILR  R180LT) 

INPOT  CORRENT  1  COMMAND  2*3  PHASE  4  TYPE  5  ACQBYRS  6 
PPARTIC  7  CP  ART  I C  S  PPARTICQ  9-19CPARTICQ  11-12 
PONCT  13  DEGREE  14-15  DOTY  16  PPDRIO  17  PPDR*1  18 
PPDR12  19  PPDR12  29  PPDR  14  21  AIPDR  22  PSYSOV  23 
PCDR17  24  PCDR18  25  PCDR19  26  PCDR29  27  PCDR21  28 
PCDR22  29  AICDR39  CSYSOV  31  LPDR25  32  LPDR26  33  LPDR27 
34  LPDR28  35  LPDR29  36  LPSYOV  37  LPAI  38  LPAPPV  39 
LCDR33  49  LCDR34  41  LCDR35  42  LCDR36  43  LCDR37  44 
LCSYOV  45  LCAI  46  DSGMOD  47  DSGMODQ  48-59  LCAPPV  51 
CMPLDSG  52  GOIDNCE  53  CDRSLIP  54  CDRSLIPQ  55-56 
POPENAI  57  COPBNAI  58  POPENAIQ  59-61  COPBNAIQ  62-64 
PPRBVAI  65  CP REV A I  66  PCLSEAI  67  CCLSEAI  68  DSGPROB  69 
VERBAL  76  «  2  CDRPREP  1  CCDRPK  2  DCDRPK  3  SLIP  4 
SLIPSOB  5-7  S)0(NLS  8-9  TRAING  19  PAQEXP  11-12 
CAQEXP  13-14  PRBV  15  CREV  16  PPERSQ  17  CPERSQ  18) 

LABEL  CURRENT- ' CORRENT  MILITARY  RANK' 

COMMAND- 'ACQUISITION  PHASE  MOST  FAMILIAR' 

PHASE- 'ACQUISITION  PHASE  MOST  FAMILIAR' 

TYPE- 'ACQUISITION  TYPE  MOST  FAMILIAR' 

ACQEYRS-' YEARS  OF  ACQUISITION  EXPERIENCE' 

PPARTIC- 'NUMBER  OP  PDRS  PARTICIPATED  IN' 

CPARTIC-' NUMBER  OF  CDKS  PARTICIPATED  IN' 

FUNCT-* CURRENT  FUNCTIONAL  AREA' 

DEGREE-' EDUCATION  TYPE 
DUTY- 'DUTY  AFSC' 

PPDR18- ' P  EVAL  TECHN ,  COST,  SCHDLE  RISK  TO  DESIGN' 
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PPDRU-'P  IV AL  DBSIGN  TO  RBQO I RIMIMT8 1 
PPDBR12"'?  IVAL  MANUP  PR0C888  RISK1 
PPD113"' P  08PXHB  INTIRFACBS* 

PPDR14-’P  SYSTEM  OVBRV X EM/TUTOR l At ' 

A1PDR"'P  R880LVB  AX8  BBPOR8  APPRVQ  PDR' 

PSY80V* '  P  PURPOSE  *  OP  NOOR8  FOR  STS  OVRVU/"OTO' 
PCDR17"'C  BVAL  TBCHN  * COST , 8CHBD0LB  TO  DESIGN* 

PCDR18-'C  BVAL0AT8  088 IQ*  TO  RBQOXRBMBNTS' 

?CDR19"'C  AS8I88  CX  T,C,8  RISK' 

PC0R2S«*C  ASSBSS  RESULTS  OP  PRODOCXBXLXTY  ANAL' 
PCDR21«'C  RIVXBtf  PRBl INXNARY  PRODOCT  SPBC' 

PCDR22-'C  REVIEW  MAJOR  DB8XGN  MODS' 

AICDR-'C  RBSOLVB  AX8  BBPORB  APPROVING  COR ' 

CSYSOV-’C  «  OP  BOORS  FOR  SYS  OVBRV XBW/TOTORXAL' 

LPDR25" ' LA8T  P  ADQTLY  BVALD  TICHM  D8GN  APPROACH' 
LPDR26*'LAST  P  ADQTLY  BVALD  T,C,S#RX8K  TO  OSQM' 

LPDR27" '  LAST  P  ADQTLY  BVALD  DSGN  TO  RIQRMNTS ' 

LPDR28" ' LAST  P  ADQTLY  BVALD  MANOPG  PR0CBS8  RISK' 

LPDR29" ' LAST  P  ADQTLY  DIF X MID  XNT8RPACI8' 

LPSYOV* ' LAST  P  I  OP  HRS  DKVTBD  TO  SYS  OVRVO/TOTO' 

LPAI" 'LAST  P  WERE  AXS  RBSLVBD  BBPORB  P  APPRVL* 

LPAPPV" ' LAST  PDR  APPROVAL  GIVIS' 

LCDR33-’LAST  C  ADQTLY  BVALD  DSGH  TO  RQORMBNTS ' 

LCDR34" ' LAST  C  ADQTLY  RBVD  DSGH  COMPATIBILITY' 

LCDR35" ' LAST  C  ADQTLY  ASSD  Cl  T,C,S  RISK'  ' 

LCDR3S" ' LAST  C  ADQTLY  ASSD  RSLTS  OP  PROBLTY  ANAL' 
LCDR37" ' LAST  C  ADQTLY  RBVD  PRBLIM  PROD  SPBC' 

LCSYOV" ' LAST  C  •  OP  HRS  DBVTD  TO  SYS  OVBRVO/TOTO' 
DSGNOD" ’ LAST  C  NBRB  MAJ  DSGN  MODS  PRBSBHTBD ' 

DSGMODQ" ' LAST  C  •  OP  MAJ  DSGN  MODS  PRESENTED' 

LCAPPV" ’ LAST  CDR  APPROVAL  GIVBN' 

CMPLDSG-'C  HOT  CHDCTD  PRIOR  TO  DBTLD  DSGN  COMPL' 
GUIDNCE-'HAVB  YOO  ANY  GOIDNCB  ON  PDR/CDR  RELBS ' 

CDRSLIP" ' LAST  CDR  SCHBDQLB  SLIPPED' 

POPBNAI • ' LAST  PDR  CONCLOSXOH  AXS  LBPT  OPIN' 

COPENAI ■ ' LAST  CDR  CONCLOSION  AIS  LBPT  OPBN' 

PPREVAI" ' LAST  PDR  AIS  PRBSBNTBD  ALREADY  EXISTED' 

CPRBVAI * ' LAST  COR  AIS  PRESENTED  ALREADY  EXISTED' 

PCLSBAI " ' LAST  P  PRBVS  AIS  CLSD  AT  COMPLN  OP  MTG' 

CCLSE A I " ' LAST  C  PREVS  AIS  CLSO  AT  COMPLN  OF  MTG' 
DSGPROB" ' LAST  C  DSGN  PROBS  COVERED  IN  BRLER  MTG' 

VERBAL** ' RQRD  TO  VBRBLISE  INDIVIDUAL  OBJS* 

CDRPREP" ' ADQUATB  TIME  TO  REV  SUPPORTING  CDR  OBJ' 
CCDRPK-'CDR  SUPPORTING  PACKAGE  MAS  COMPLETE' 

DCDRPK"'CDR  SUPPORTING  PKG  NAS  DELVRD  ON  SCHBD ' 

SLIP" 'LAST  PROG  DSGN  AND  DVLMNT  SCHDLB  SLPBD' 

TRA I NG- 'PDR/CDR  TRAINING  MOULD  BE  USEFUL' 

CAQBXP* ' *  OP  MO  OP  ACQ  EXPERIENCE  RQRD  FOR  CDR' 

PREV" ' LAST  P  ADQTLY  TO  ENTER  NEXT  ACQ  PHASE' 

CREV" ' LAST  C  ADQTLY  TO  ENTER  NEXT  ACQ  PHASE' 

PPBRSQ-' NUMBER  OP  AP  PARTICPANTS  AT  LAST  PDR' 

CPBRSQ- ' NUMBER  OP  AP  PARTICIPANTS  AT  LAST  CDR' 

FORMAT  PPDR10  PPDR1I  PPDR12  PPDR13  PPDR14  PCDR17  PCDRI8 

PCDR19  PCDR20  PCDR21  PCDR22  LPDR25  LPDR26  LPDR27  LPDR28 
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LPDR29  LCDR33  LH)R34  LCDR35  LCDR36  CDKR37  LIKBR. 
GOXDNCB  VBRBAL  PPARTXC  CPARTIC  YBSNO.  AIPDR  AXCDR  LPAI 
LCAI  D80M0D  CMP LOSS  CDRSLIP  POPBNAX  COPBNAI  PPRBVAX 
CPREVAI  PCL8BAI  CCL8BAX  D SOP ROB  CDRPRBP  CODRPK  DCDRPK 
SLIP  TRAINS  PRSV  CRIV  PBNSB.  P8I80V  CSV 80V  LP8Y0V 
LCSYOV  OVtRVO •  PPSR8Q  CPBRSQ  QTYPBRS,  PAQBXP  CAQIXP 
OJT.  LPAPPV  LCAPPV  APPROVAL.! 

PROC  PRIQl 

TABLB8  CORRBNT--  ACQBYRS ? 

TABLB8  PONCT— CDR8LIP; 

TABLB8  POPBMAJ } 

TABL88  COPBNAI j 
TABLB8  PPRIVAI—  SLIP; 

TABL88  TRAINS— CAQBXP! 

TABLBS  PPARTXC* (PPDR10- -PPDR13) f 
TABLES  CPARTIC* (PCDR17-- PCDR22 ) f 
TABLB8  DUTY*  (PPDR18  — PPDR13)  ! 

TABLBS  DOTY*  (PCDR17— PCDR22)  J 

TABLB8  PPARTXC*  (PP0R18— PP0R13)  ♦  (LPOR25— LPDR2P)  } 
TABLBS  CPARTIC*  (PC0R17— PCDR22)  *  (LCOR33— LCOR37)  ; 
TABLBS  LPAI*LPAPPV| 

TABLBS  PPRBVAI*PCLSSAI) 

TABLBS  CPRSVAI*CCLSBAX} 

TABLBS  CDRSLXP*SLXP  /  CHISQj 
TABLBS  PPAKTIC*PRBV| 

TABLBS  CPARTIC*CRBV; 

TABLBS  PPARTIC*DUTY*PSYSOV| 

TABLBS  CPARTIC*DUTY*CSYSOV| 

TABLBS  PSYSOV*LPSYOV ; 

TABLES  CSYSOV*LCSYOV } 

TABLBS  PPARTIC*DOTY*AIPDR; 

TABLBS  CPARTIC*DOTY*AICDR; 

TABLBS  CPARTIC*DOTY*CNPLDSG; 

TABLBS  DOTY*GUIDNCEj 
TABLBS  DUTY*TRAXNC; 

TABLBS  PPARTIC*OOTY*PAQEXP ; 

TABLBS  CPARTIC*DUTY*CAQEXP; 

TABLES  DOTY*PPARTIC; 

TABLBS  DUTY*CPARTICj 
PROC  CORR; 


109 


Appendix  Cs  Selected  Comments 


Question  10: 


"What  about  supportability." 


Question  14* 

"Should  not  have  to  be  this  way  but  is  a  very  necessary 
step." 


Question  15: 


"Depends  on  action  items." 

"Often  you  discover  one  or  two  that  were  inappropriate  at 
the  time  they  were  given  for  the  stage  of  deve’’  opment ,  these 
can  be  passed  on  to  CDR  or  eliminated. 

"Some  M's  are  more  important  than  others. 


Question  16: 

"Stupid  answers,  depends  upon  size  of  system." 

"Depends  on  the  size  of  the  design.  This  should  be 
expressed  as  a  %  of  PDR  time." 

"This  is  very  dependent  upon  system  complexity." 

"Depends  on  the  size  of  the  program." 

"Strong  function  of  system  complexity  and  personnel 
turnover . " 

"No  more  than  a  half  a  day's  effort." 

"Varies  depending  on  complexity  of  program,  enough  info 
should  be  presented  so  that  praticipants  understand  the 
function  and  areas  where  technical  concern  is  already 
apparent." 
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"None  -  too  much  tiro*-  is  spent  preparing  strap  hangers  who 
usually  up  to  this  point  in  time  have  not  had  active  roles 
in  the  project.  We  waste  too  much  time  on  dog  and  ponies. 
This  costs  tax  payers.  Before  you  have  the  SPO  all  pre¬ 
coordination  and  training  should  have  been  performed. 


Question  19: 

"What  about  supportability. " 


Question  21: 

"Not  at  a  CDR 1  Spec's  should  be  well  entrenches  and 
understood  by  participants  at  this  point,  assumly  no  "new" 
players  involved." 


Question  22: 

"If  this  occurs  you  have  a  second  PDR  in  my  book." 


Question  23: 


"Some  minor  items  could  be  left  to  a  later  date  but,  if  any 
chance  they  could  cause  serious  problems  they  should  be 
resolved  before. 


Question  24: 

"Time  spent  depends  upon  the  system  in  which  the  review  is 
created  to  support.  Different  projects  require  different 
CDRs." 

"However,  depends  on  the  system  and  its  complexity." 

"No  more  than  half  a  day's  effort." 

"None  -  too  much  time  is  spend  preparing  strap  hangers  who  t. 
usually  up  to  this  point  in  time  have  net  had  active  roles 
in  the  project.  We  waste  too  much  tidme  on  dog  and  ponies. 
This  costs  tax  payers.  Before  you  have  the  SPO  all  pre- 
coordination  and  training  should  have  been  performed. 


Question  41: 

"This  was  the  only  review  (PDR  orCDR)  that  I  was  involved  in 
that  was  desapproved,  though  others  I  thought  should  have 
been." 


Question  42: 


"At  least  the  major  system  CIs." 

"Incremental  are  necessary  on  complex  systems." 


Question  43: 

"Everyone  including  myself  has  no  knowledge  of  what  was 
going  to  happen  at  the  PDR." 

"Reviews  should  be  used  to  clear  up  problems  and  to  provide 
last  minute  design  information.  They  should  not  be  used  as 
a  tutorial  (this  wastes  time) .  You  should  conduct  yourself 
as  a  professional.  Getting  mad  or  loud  is  inappropriate. 

"Learning  from  early  reviews  how  to  conduct  (PDR,  CDR ,  FCA, 
PC AS) .  Last  reviews  we  briefed  our  people  how  to  identify 
action  and  ask  for  resolution.  This  was  absolutely 
essential  when  we  had  to  disapprove  the  CDR." 


Question  49: 

"But  this  was  held  after  PDR/CDR. " 

Question  50: 

"Data  not  delivered  prior  to  CDR." 

"Mo  real  preparation  on  my  part." 

"The  package  of  software  documentation  that  arrived  was 
incomplete;  it  was  also  a  little  late." 

"The  time  required  was  too  short;  data  was  usually  late 
and/or  not  sufficient." 

"Never  enough  time." 

"Engineering  drawings  lacked  adequate  detail." 
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"Time  constraints  -  politically,  CDR  had  to  be  held  then." 
"Review  time  adequate;  comments  not  addressed  prior  to  CDR." 
"Data  packages  were  incomplete." 

"CDR  packages  do  not  arrive  until  2-3  weeks  prior  to  CDR." 

"10  days  is  insufficient  to  review  a  thousand  pages  of 
documents  and  drawings." 

"This  is  extremely  important  -  a  PDR/CDR  dry. run  to  the  AF 
program  manager/office  should  occur  at  least  a  few  weeks 
prior  to  the  review  to  approve  content,  format  and 
delivery. " 

"Delivered  at  CDR." 

"Always  said  they  would  get  back  to  us." 

"Insufficient  time  between  getting  documentation  and 
PDR/CDRS." 

"Contractor  documents  were  later  than  required  and  at 
meeting  he  presented  new  data  also." 

"Delay  in  delivery  and  poor  quality  of  CDR  package  precluded 
adequate  review." 

"Definitely  a  problem." 

"Far  too  much  to  review." 

"The  packages  always  seem  to  be  late  creating  a  time 
crunch. " 

"Submitted  only  5  days  prior." 

"CDR  supporting  data  was  not  received  in  time  to  submit 
comments  to  the  contractor  prior  to  CDR." 

"Support  packages  was  delivered  one  week  in  advance. 

Package  consisted  of  three  thousand  plus  pages." 

"The  scheduled  time  of  review  was  short  because  the 
information  from  the  contractor  was  late." 

"Not  enough  time  to  review  but  comments  were  addressed." 

"Data  delivered  late  and  CDR  proceeded  to  stay  on  schedule." 

"Held  the  review  with  out  properr  review  to  preserve  the 
project  schedule." 
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"All  documents  were  presented  but  as  a  joint  program  review 
was  not  complete  until  after  the  event." 

"«o  data  provided  to  COR." 

"Contractor's  documentation  abilities  were  very  bad." 
"Scheduled  pressure." 


"Review  time  and  comment  review  cycle  was  too  short. 
Approximately  1  month  minimum  is  required  for  satellite 
level  reviews." 

"Contractor  tried  to  press  ahead  without  SPO  approval.” 

"Data  is  never  reviewed  well  enough  by  meeting  attendees 
prior  to  meeting.  Design  reviews  are  rarely  a  "review"  of 
anything  -  they're  a  presentation  of  design  that  most 
attendees  did  not  do  their  homework  to  understand." 

"Rarely  do  you  get  the  material  prior  to  the  review  (like 
you  should)." 

"Packages  arrived  too  close  to  CDR  date  to  permit  adequate 
review.  Consequently  questions  could  not  be  asked  of 
contractor  prior  to  the  CDR." 


Question  51: 

"Updated  drawings  (in  the  proper  format)  and  specifications 
were  very  late." 

"Some  of  the  data  wasn't  available  until  the  day  of  the 
review." 

"Reviewed  material  prior  to  CDR,  but  time  limited  contractor 
responses  prior  to  CDR." 


Question  52: 

"Caused  by  Government  due  to  lack  of  funding." 


Question  54: 

"This  depends  on  many  factors.  It's  the  best  way  to  learn 
about  one's  job." 
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"Hot  important:  Can  ba  a  learning  experience." 


Quaation  55: 

"PDR  -  Unknown." 

"COR  -  Spool float ions." 

"COR  -  Can't  remember  the  manual  my  boss  recommended. " 

"1  hava  not  found  such  guide." 

"Ravlawing  tha  contractor's  data  packages  are  the  most 
important  guide  for  knowing  what  areas  to  focus  on." 

"PDR  -  a  chairman  who  knows  his  job  and  has  chaired  several 
PDRs  before  (AF) . " 

"COR  -  a  chairman  who  knows  his  job  and  has  chaired  several 
CDRs  before  (AF)." 


Questions  10-14: 

"Represents  activities  which  should  occur  before  PDR.  If 
you  don't  know  the  answers  to  these  questions  before  you 
arrive  at  the  PDR,  you  have  no  business  being  there." 


General  Comments; 


"PDR  and  CDR  review  procedures  and  expected  conduct  need  to 
be  formally  taught  1  We  also  need  to  put  emphasis  on 
meaningful  participation  and  homework  by  those  attending.  • 
Design  reviews  are  serious  business  and  a  place  for  real 
work  by  involved  people.  Too  often  they  are  viewed  as  a 
source  of  TDY  funding  and  free  donuts  for  the  masses." 
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This  investigation  examined  the  preparedness  of  Preliminary  Design 
Review  (PER)  and  Critical  Design  Review  (CDR)  participants.  Background 
and  opinion  data  were  gathered  from  junior  and  senior  program  managers 
and  development  engineers  in  order  to  assess  their  perception  of  PER  and  CER 
purposes,  ef fectivenosa/ef f iciandes ,  training  and  guidance. 

The  analysis  was  aooorplished  fay  sending  a  survey  instrument  to  a 
sampling  of  progran  managers,  and  development  engineers  throughout  the  Air 
Force  population  within  the  boarders  of  the  U.S.A. 

The  respondents  tended  to  be  in  agreement  with  the  FER/CDR 
purposes  stated  in  nil  Std  1521. 

The  analysis  revealed  that  most  PER/CDRs  are  rot  as  effective 
as  they  could  be.  The  primary  reason  is  the  lade  of  knowledge  on  what 
should  be  aooenplished  by  the  participants. 

Most  respondents  claimed  self  teaching  as  the  method  of  learning 
their  preparation  for  PERAXRs.  Over  86.2  percent  of  all  respondents 
felt  initial  training  would  be  useful  and  the  majority  indicated  6  to  12 
nonths  of  acquisition  experience  should  be  required  before  participating 
in  a  PDR/tER.  The  rest  important  guide  for  PER/CDR  preparation  reoermended 
by  the  respondents  with  previous  PER/CER  experience  wes  the  Mil  Std  1521 
and  Defense  System  Management  College  (DSMQ  Systan  Engineering  Management 
Guide. 
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